Why Amazon hates 'human-in-the-loop' AI governance

Humans, like AI agents and systems, are non-deterministic. Neither can be guaranteed to produce the same output given the same input twice.

“We know how humans fail,” Brandwine said. “We're comfortable with it. So human-in-the-loop isn’t necessarily the gold standard.”

For years, vendors have told companies that the solution for dealing with any automated system was to put a human in the loop.

Normalization of deviance

happens when people in an organization take shortcuts, or don’t follow the established procedures or standards, and sometimes it occurs over years. As long as nothing catastrophic happens, this deviant behavior becomes the norm.

Humans build LLMs and AI systems, and having a “human-in-the-loop” ensures that a person reviews the AI’s output and approves (or not) any actions before the AI performs them.

After a repeated false alarms, humans just ignore the alarms.

Big tech pulls the human-in-the-loop

"It is very clear that we have moved from a human-led defense strategy, to a human-in-the-loop defense strategy, to an AI-led defense strategy that's overseen by humans,"

loop learning instead of having a human check an AI’s output at every step.

(Secret) keys to the kingdom

At Amazon, all of the agents have independent identities assigned to them, we’re told.

goal-seeking behavior- when a person asks an agent to do a specific task and the agent becomes laser-focused on just one action to achieve this goal, ie, deleting the database.

Telling the agent why it doesn’t have permission to do something tends to produce a better outcome,

The person that wants to run the agent wants to give the agent many permissions because that makes the agent more powerful,