We’re excited to release a new AI governance research agenda from the MIRI Technical Governance Team. With this research agenda, we have two main aims: to describe the strategic landscape of AI development and to catalog important governance research questions. We base the agenda around four high-level scenarios for the geopolitical response to advanced AI development. Our favored scenario involves building the technical, legal, and institutional infrastructure required to internationally restrict dangerous AI development and deployment (which we refer to as an Off Switch), which leads into an internationally coordinated Halt on frontier AI activities at some point in the future. This blog post is a slightly edited version of the executive summary.

We are also looking for someone to lead our team and work on these problems, please reach out here if you think you’d be a good fit.

The default trajectory of AI development has an unacceptably [...]

---

Outline:

(04:44) Off Switch and Halt

(07:33) US National Project

(09:49) Light-Touch

(12:03) Threat of Sabotage

(14:23) Understanding the World

(15:07) Outlook

---

First published:
May 1st, 2025

Source:
https://www.lesswrong.com/posts/WkCfvqyjCzvRrwkaQ/ai-governance-to-avoid-extinction-the-strategic-landscape

---

Narrated by TYPE III AUDIO.

---

Images from the article:

A rough tree of the scenarios discussed in this report and how AI governance paradigms may evolve. This diagram assumes catastrophe is avoided at each step, so the myriad failures are omitted. This is simplified, and one could draw many more connections between the scenarios.

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

“AI Governance to Avoid Extinction: The Strategic Landscape and Actionable Research Questions” by peterbarnett, Aaron_Scher

LessWrong (30+ Karma)