Stanford CS229 Machine Learning | Spring 2026 | Lecture 7: Neural Networks 1 (Architecture)

Deep learning relies on supervised learning frameworks where non-linear models, specifically neural networks, map inputs to outputs. Training these models involves minimizing a loss function—such as mean squared error for regression or cross-entropy for multi-class classification—using stochastic gradient descent (SGD). SGD approximates the full gradient by sampling individual examples, which is computationally efficient for large datasets. Neural networks utilize non-linear activation functions like ReLU to capture complex patterns, while architectures like ResNet incorporate residual connections to simplify optimization by modeling differences rather than absolute values. Furthermore, techniques like layer normalization and RMSNorm stabilize training by maintaining consistent scaling across layers, preventing numerical instability. These components collectively enable the construction of deep, high-dimensional models capable of learning intricate representations from vast amounts of data.