This is a link post.

At last. Highlights: some details on Preparedness Framework evals + evals by METR and Apollo.

Preparedness framework evaluations

You should follow the link and read this section.

Brief comments:

Cyber: the setup sounds good (but maybe substantially more powerful scaffolding/prompting is possible). Separately, I wish OpenAI shared the tasks (or a small random sample of them) or at least said more about where they came from. (Recall that DeepMind shared CTF tasks.)
Bio uplift: GPT-4o clearly boosts users on biological threat creation tasks — OpenAI doesn't say that but shows a graph. (It continues to be puzzling that novices score similarly to experts.) (I kinda worry that this is the wrong threat model—most bio risk from near-future models comes from a process that looks pretty different from a bigger boost to users like these—but I don't have better ideas for evals.)
Persuasion: unclear whether substantially [...]

---

Outline:

(00:16) Preparedness framework evaluations

(01:34) Third party assessments

(03:26) Context

---

First published:
August 8th, 2024

Source:
https://www.lesswrong.com/posts/nBtryzLzLnsHEiKRn/gpt-4o-system-card

---

“GPT-4o System Card” by Zach Stein-Perlman

LessWrong (30+ Karma)