Long-Term Risk: AGI, Recursive Self-Improvement, and the Control Problem

If AI can improve itself, do humans remain in the loop, and how would we know if we'd lost control?

1. Current Industry Evidence

  • Most of Anthropic's code is written by AI.
  • Currently 100% better every 4 months.
  • Feedback loop or just normal efficiency improvements?

    • 2. The Three Scenarios

    1. The Stall

    AI capability growth decelerates and eventually plateaus.

    2. Compounding Gains

    Improvements continue at approximately their current pace.

    3. Recursive Self-Improvement (RSI)

  • AI systems capable enough to meaningfully improve AI systems is used to improve itself (or its successors), producing a more capable system that can make further improvements, which produces an even more capable system, and so on.
  • AI safety researchers treat the RSI scenario seriously for 2 reasons-
  • Magnitude- RSI could produce AI far more capable than humans in essentially every domain in a relatively short time.
  • Asymmetry- Preparing for RSI would be much less constly than being unprepared.

    • 3. The Alignment Problem

  • The challenge of ensuring that AI systems pursue goals that are actually beneficial to humans.
  • Nick Bostrom proposed the orthogonality thesis: intelligence and goals are orthogonal. AI could have goals that are completely alien to human values.
  • The instrumental convergence thesis (Bostrom, Turner) holds that many final goals tend to generate the same set of sub-goals: self preservation, resource acquisition, cognitive enhancement, and goal-content integrity (don't change the goal)

    • Stuart Russell's Response: Human Compatible AI

  • Russell argues in his book Human Compatible that the alignment problem points to a fundamental error in how AI is currently designed.
  • AI systems should not have fixed objectives.
  • Russell's framework is an attempt to build deference into the design of AI systems.
  • 1. Have deep uncertainty about human preferences
  • 2. Act to benefit humans as inferred from evidence about human behavior and values
  • 3. Defer to humans in cases of uncertainty, and allow themselves to be corrected
    • It tries to figure out what humans want and act to help them get it. It would not resist being switched off

    4. Expert Disagreement and What To Do With It

    AGI- Artificial General Intelligence

    Philosophy can offer:
  • Clarification- The alignment problem requires multiple premises (orthogonality thesis + instrumental convergence + capability growth) to be jointly true for its worst-case scenarios
  • Thinking about obligations- . If there is a probability of a catastrophic AI outcome, what obligations does this generate?
  • Evaluating proposed solutions (Russell's human-compatible AI, proposed global pauses, international AI treaties, and interpretability research)
  • Resisting premature certainty- Technology forecasting suggests that the optimists and the pessimists tend to be wrong about timelines and mechanisms.

    • 5. What Should We Do Now?

  • Take action now to reduce risk even under uncertainty.
  • AI's potential benefits are so large that slowing development carries its own massive moral costs.
  • There is a compelling argument that decisions of this magnitude must be made democratically.
  • democratic processes are slow and AI capability development is fast