The Real Reason Huge AI Models Actually Work [Prof. Andrew Wilson]

The discussion centers on challenging conventional wisdom in machine learning, particularly around the bias-variance tradeoff and the role of model complexity. Andrew Wilson, a professor at NYU, argues against the necessity of a bias-variance trade-off, advocating for expressive models with soft inductive biases that adapt to both small and large datasets. He shares insights on deep learning's relative universality and its effectiveness in representation learning, highlighting the importance of scale in achieving good generalization through a simplicity bias. The conversation explores misconceptions in understanding generalization, such as the belief that models should change based on available data points, and delves into the mysteries behind the simplicity bias at scale, touching on loss landscapes and compressibility. Wilson also touches upon the potential for AI to discover new scientific theories.