Just an interesting philosophical argument

1. Physics

Why can an ML model learn from part of a distribution or data set, and generalize to the rest of it? Why can I learn some useful heuristics or principles in a particular context, and later apply them in other areas of my life?

The answer is obvious: because there are some underlying regularities between the parts I train on and the ones I test on. In the ML example, generalization won't work when approximating a function which is a completely random jumble of points.
Also, quantitatively, the more regular the function is, the better generalization will work. For example, polynomials of lower degree require less data points to pin down. Same goes for periodic functions. Also, a function with lower Lipschitz constant will allow for better bounding of the values in un-observed points.

So it must be that the variables we [...]

---

Outline:

(00:07) 1. Physics

(01:31) 2. Anthropics

(02:03) 3. Dust

---

First published:
February 20th, 2024

Source:
https://www.lesswrong.com/posts/NwDvmodb2CcuysQxW/why-does-generalization-work-1

---

Narrated by TYPE III AUDIO.

“Why does generalization work?” by Martín Soto

LessWrong (30+ Karma)