A pdf version of this report is available here.

Summary.

In this report we argue that AI systems capable of large scale scientific research will likely pursue unwanted goals and this will lead to catastrophic outcomes. We argue this is the default outcome, even with significant countermeasures, given the current trajectory of AI development.

In Section 1 we discuss the tasks which are the focus of this report. We are specifically focusing on AIs which are capable of dramatically speeding up large-scale novel science; on the scale of the Manhattan Project or curing cancer. This type of task requires a lot of work, and will require the AI to overcome many novel and diverse obstacles.

In Section 2 we argue that an AI which is capable of doing hard, novel science will be approximately consequentialist; that is, its behavior will be well described as taking actions in order [...]

---

Outline:

(06:22) Acknowledgments

(06:51) How to read this report

(08:23) Related work

(10:40) Section 1: Useful tasks, like novel science, are hard

(12:14) Impactful novel science takes a lot of work

(13:32) Outcome-oriented tasks

(14:54) Novelty and diversity of obstacles

(16:51) AI alignment research is hard

(18:56) Stopping misaligned AI deployment seems to require powerful aligned AI

(19:32) Conclusion

(20:22) Section 2: Being capable of hard tasks implies approximate consequentialism

(21:00) Future outcomes as goals

(22:09) Formalizing consequentialist goals

(23:00) Non-consequentialist constraints

(24:11) Combined goals

(25:31) Why consequentialist goals are a necessary part of powerful AI

(25:37) Robustness to diverse obstacles is driven by consequentialism

(29:17) Shallow and deep constraints

(30:53) Approximation

(33:10) Section 3: Hard tasks require learning new things

(33:44) The AI will need to learn new things

(34:06) Learning facts

(34:51) Learning skills

(36:32) Self-directed learning

(37:01) Human-directed learning is a big efficiency hit

(38:05) Indirect self-directed learning

(38:43) Useful versus safe tradeoff

(39:36) Examples

(41:37) Conclusion

(42:28) Section 4: Behavioral training is an imprecise way to specify goals

(45:42) Unstable goals

(45:56) Beliefs and goals can be mixed together

(47:57) Outer shell non-consequentialist constraints

(50:38) Changing goals over time

(53:13) Out-of-distribution generalization

(53:18) OOD generalization difficulty

(54:50) Instrumental goals as terminal goals

(56:05) Deliberate deception

(57:00) Badly designed training incentives

(57:59) Conclusion

(58:57) Section 5: Control of misaligned AIs is difficult

(01:02:11) Misaligned goal-directed AI won’t cooperate with humans

(01:02:59) Eliciting useful capabilities via outer training loop

(01:03:52) Difficulties

(01:07:05) Difficulty of escape compared to routine problem-solving

(01:09:36) Difficulty of sabotage and delay

(01:11:49) Transparency and honesty

(01:14:42) Eliciting dangerous capabilities

(01:15:32) Difficulties

(01:17:08) AI assisted monitoring doesn’t change the picture much

(01:18:37) Monitoring with powerful AI systems

(01:22:04) Monitoring with weak AI systems

(01:24:11) Conclusion

(01:25:27) Section 6: Powerful misaligned AI would be bad

(01:26:23) A misaligned AI achieving its goals would be bad for humans

(01:26:56) Instrumental convergence toward gaining control of limited resources

(01:27:34) Ambitiousness

(01:28:46) Consequences of misspecified goals

(01:33:20) Why would a powerful AI be capable of defeating humanity, given moderate levels of freedom?

(01:35:25) Conclusion

(01:37:02) Requests for AI developers

(01:39:31) Definitions

(01:41:15) Appendix: Argument graph

(01:47:55) Appendix: Attack surfaces

(01:52:20) Appendix: 4-hour research assistant

The original text contained 40 footnotes which were omitted from this narration.

---

First published:
January 26th, 2024

Source:
https://www.lesswrong.com/posts/GfZfDHZHCuYwrHGCd/without-fundamental-advances-misalignment-and-catastrophe

---

Narrated by TYPE III AUDIO.

“Without fundamental advances, misalignment and catastrophe are the default outcomes of training powerful AI” by Jeremy Gillen, peterbarnett

LessWrong (30+ Karma)