What is a Transformer? (Transformer Walkthrough Part 1/2)

In this two-part video series, Neel introduces transformers, the architecture behind modern language models, and aims to provide a deep understanding of how they work by coding one from scratch. Neel, a researcher in mechanistic interpretability, explains why understanding the internals of these models is crucial, especially as they achieve human-level language capabilities. The first part focuses on conceptually explaining transformers, their components, and their purpose, targeting those new to the topic but familiar with neural networks. Neel discusses the inputs (tokens) and outputs (logits) of a transformer, emphasizing that transformers are sequence modeling engines that process information in parallel at each sequence position and use attention to move information between positions. The tutorial also covers tokenization, embeddings, layer normalization, and positional information, providing a comprehensive overview of transformer architecture.