ARC's current research focus can be thought of as trying to combine mechanistic interpretability and formal verification. If we had a deep understanding of what was going on inside a neural network, we would hope to be able to use that understanding to verify that the network was not going to behave dangerously in unforeseen situations. ARC is attempting to perform this kind of verification, but using a mathematical kind of "explanation" instead of one written in natural language.

To help elucidate this connection, ARC has been supporting work on Compact Proofs of Model Performance via Mechanistic Interpretability by Jason Gross, Rajashree Agrawal, Lawrence Chan and others, which we were excited to see released along with this post. While we ultimately think that provable guarantees for large neural networks are unworkable as a long-term goal, we think that this work serves as a useful springboard towards alternatives.

In this [...]

---

Outline:

(01:48) Formal verification for neural networks

(04:24) Heuristic explanations

(07:00) Surprise accounting

(09:06) Worked example: Boolean circuit

(09:39) Brute-force

(10:12) Final OR gate

(11:05) Pattern of NOT gates

(13:03) Pattern of AND and OR gates

(15:11) Discussion

(16:26) Conclusion

The original text contained 1 footnote which was omitted from this narration.

---

First published:
June 25th, 2024

Source:
https://www.lesswrong.com/posts/SyeQjjBoEC48MvnQC/formal-verification-heuristic-explanations-and-surprise

---

Narrated by TYPE III AUDIO.

“Formal verification, heuristic explanations and surprise accounting” by paulfchristiano

LessWrong (30+ Karma)