Andrej Karpathy

How I use LLMs

This podcast is a monologue by Andrej Karpathy providing a comprehensive overview of Large Language Models (LLMs) and their applications in 2025. He begins by discussing the evolution of LLMs from ChatGPT's initial release to the current diverse ecosystem of providers and models, emphasizing the importance of understa...

Deep Dive into LLMs like ChatGPT

This podcast provides a detailed explanation of how large language models (LLMs) like ChatGPT are built and function. The speaker walks the listener through the three main stages of LLM training: pre-training (using internet data), supervised fine-tuning (creating conversational datasets), and reinforcement learning (...

Founding fathers on today's America

This episode explores how the Founding Fathers of the United States might perceive contemporary America across various domains. Against the backdrop of the original constitutional framework, the podcast analyzes the expansion of federal power, the evolving relationship between the executive and legislative branches, an...

Let's reproduce GPT-2 (124M)

In this podcast episode, listeners are taken on an enlightening journey as the host walks through the detailed process of recreating the GPT-2 model with 124 million parameters from the ground up using PyTorch. The discussion covers key aspects like model architecture, loading weights, optimization techniques, and ways...

Let's build the GPT Tokenizer

This podcast episode explores the importance of tokenization in large language models. The speaker highlights that many issues faced by these models, such as difficulties in spelling tasks, processing non-English languages, and limitations in performing arithmetic, often stem from tokenization. The episode discusses th...

[1hr Talk] Intro to Large Language Models

This podcast episode explores the world of large language models, focusing on the LLAMA270B model released by Meta.ai. The speaker discusses the accessibility and self-contained nature of the model, as well as the process of acquiring and training the parameters. The episode delves into the workings of neural networks,...

Let's build GPT: from scratch, in code, spelled out.

This podcast episode explores the development and implementation of ChatGPT, a powerful AI system based on the transformer architecture. The episode covers topics such as the Tiny Shakespeare dataset used for training, language modeling, tokenization strategies, and the process of re-representing a dataset. It also del...

Building makemore Part 5: Building a WaveNet

This episode explores the development and improvement of a character-level language model, MakeMore, focusing on architectural complexities and practical implementation challenges. Against the backdrop of a previous simpler model, the speaker introduces a deeper architecture inspired by WaveNet, a speech synthesis mod...

Building makemore Part 4: Becoming a Backprop Ninja

This episode explores the implementation of manual backpropagation in neural networks, emphasizing its importance despite the prevalence of automated tools like PyTorch's Autograd. Against the backdrop of readily available Autograd engines, the lecture advocates for understanding backpropagation's internals to debug an...

Building makemore Part 3: Activations & Gradients, BatchNorm

This episode explores the intricacies of neural network initialization and optimization, focusing on the importance of understanding activation and gradient behavior during training. Against the backdrop of implementing a multilayer perceptron for character-level language modeling, the speaker identifies initial probl...

Building makemore Part 2: MLP

This episode explores the implementation of a multilayer perceptron model for character-level language modeling, building upon a previous bigram model. Against the backdrop of the limitations of bigram models in handling larger contexts, the speaker introduces a multilayer perceptron approach inspired by the Ben-Juatt...

The spelled-out intro to language modeling: building makemore

This episode explores the creation of "makemore," a character-level language model built step-by-step. The speaker introduces makemore's functionality, demonstrating its ability to generate unique, name-like strings from a large dataset of names. Against this backdrop, the core concept of character-level language mod...

Stable diffusion dreams of psychedelic faces

Stable diffusion dreams of steampunk brains

The spelled-out intro to neural networks and backpropagation: building micrograd

This episode explores the inner workings of neural network training, focusing on the creation of micrograd, a scalar-valued autograd engine designed for pedagogical purposes. Andrej details how micrograd facilitates the understanding of backpropagation, the core algorithm for tuning neural network weights, by building ...

Stable diffusion dreams of "blueberry spaghetti" for one night

Stable diffusion dreams of steam punk neural networks