Stanford CS336 Language Modeling from Scratch | Spring 2025 | Scaling laws 1

The podcast focuses on scaling laws in machine learning, particularly for large language models (LLMs). It begins by framing the challenge of building the best open-source LLM with limited resources, emphasizing the need to innovate rather than just copy existing models. The discussion covers the history and background of scaling laws, highlighting their grounded nature and evolution from theoretical machine learning to empirical applications. It explores data scaling, model scaling, and the interplay between data, model size, and compute, including the Chinchilla scaling laws, and addresses practical engineering decisions like hyperparameter tuning, architecture selection, and resource allocation, and the trade-offs between model size and data set size.