Some thoughts on the Sutton interview

Dwarkesh Patel reflects on his interview with Richard Sutton, focusing on Sutton's "Bitter Lesson" essay and its implications for AI development. Patel interprets Sutton's argument as advocating for AI techniques that effectively leverage compute, criticizing current LLMs for inefficient compute usage, reliance on human data, and lack of true world models. Patel disagrees with Sutton's stark distinctions between LLMs and true intelligence, arguing that imitation learning and RL are complementary. He uses the analogy of fossil fuels to explain how pre-training data can be a crucial stepping stone to AGI, even if not the ultimate solution. He also suggests that continual learning could be integrated into LLMs through techniques like supervised fine-tuning as a tool call, and concludes by noting that while Sutton's vision may not be the immediate path to AGI, his critiques highlight important gaps in current models, such as sample efficiency and dependence on human data.