AI Agents and Autonomous Decision Making

An AI agent is different because it perceives its environment, makes decisions, and acts without waiting for human approval at each step.

1. What Is an Agent?

perceive/decide/act/observe loop-
  1. Perceive: agent receives input (task description, real-time data, the results of previous actions, user instructions)
  2. Plan
  3. Act
  4. Observe: The agent observes the result of its action.
  5. Repeat: agent uses what it observed to update its plan and act again

Humans in the loop

  • Human-in-the-loop- fully supervised; agent is an assistant
  • Human-on-the-loop- supervised automation; human monitors while agent acts
  • Human-out-of-the-loop- fully autonomous; human sets goals at the start and evaluates results at the end.

    • 2. The Responsibility/Accountability Gap

    you are responsible for an outcome if
  • You caused it (in a sufficiently direct way)
  • You did so through a voluntary action
  • You knew or should have known that your action would lead to that outcome.

    • delegation does not eliminate responsibility

  • The causation problem- decisions were not reviewed by any human before they were executed.
  • The knowledge problem- agent was trained to make the best recommendation based on available evidence
  • The many hands problem
  • The autonomy problem- accountability gap: harm occurs, people are affected, but there is no clear answer to the question of who is responsible.

    • 3. Human-in-the-Loop Requirements

    Industry Pushback

    Objections

  • Efficiency
  • Quality same as efficiency?
  • Rubber stamp problem
  • Liability shifting- If the AI makes the decision, it is harder to sue the company.

    • Evaluating the Debate

  • The case for human-in-the-loop is, at its core, a case for maintaining the existing structure of moral responsibility.
  • mandating human-in-the-loop requirements may harm the people it is intended to protect.
  • Alternatives to "a human must be responsible" might be more appropriate when AI agents are involved.

    • 4. Philosophical Frameworks for Evaluating Delegation

    The Deontological Analysis

  • Deontological ethics, or deontology, is a moral theory stating that the rightness or wrongness of an action depends on whether it follows a set of rules and duties, rather than on its consequences
  • The Kantian framework would advocate for a right to human review for consequential decisions

    • The Virtue Ethics Analysis

    If AI systems increasingly make the difficult calls, humans may lose the practical wisdom that comes from being responsible for consequential judgments.

    5. Where Is the Line?

  • Stakes- A typo correction needs no human review. A medical diagnosis that determines treatment does.
  • Reversibility
  • Accountability- If no human is accountable and no recourse exists for people harmed, this is a strong argument for human involvement.
  • Track record
  • Rubber stamp problem- mandating human-in-the-loop may create false assurance without real accountability.