The Autopilot With No Pilot
An agent is handed a business goal and write access to a live system, then left to take consequential actions on its own — no approval gate, and no answer to “what happens when it’s wrong?”
What it is
An agent gets an open-ended objective — “grow revenue,” “keep inventory balanced,” “clear the queue” — and real, write-level access to a production system. Then it’s set running on a schedule or a loop. Nobody decided which of its actions need a human to look first. The first time it’s wrong, it’s already wrong in production, in front of customers, at machine speed.
How it happens
The demo is intoxicating. The agent reads the data, forms a plan, takes the action, and reports back — end to end, nobody in the middle. In a sandbox that’s magic. The leap that feels tiny and isn’t: pointing that same loop at the live system. “It worked in the demo” quietly becomes “it’s allowed to change prices.” The autonomy was the impressive part, so the autonomy is the part nobody wants to constrain.
Warning signs
- The agent can take an action a human can’t easily undo — change live pricing or content, message customers, move money, mutate inventory — with no approval step in front of it.
- Nobody can answer “what does it do when it’s wrong?” with anything but “we’d notice.”
- “We’d notice” depends on a human watching a dashboard at the exact moment the agent acts — which is to say, at 2am, nobody.
- The objective is open-ended (“increase conversion”) rather than bounded (“propose three variants for review”).
- There’s no kill switch — or there is one, and nobody has ever pulled it to confirm it works.
Why it’s dangerous
Autonomy multiplies both the upside and the blast radius. A human making a mistake makes it once, slowly, and usually catches it. An agent in a loop makes it across a thousand records before lunch, confidently, and the metric it was optimizing can move in the exact wrong direction while every log line says “success.” The cost of a wrong autonomous action isn’t the action — it’s the action times the number of times it ran before anyone looked, on a system whose entire selling point is that nobody has to look.
The AI-era hinge: the action used to be the slow part. Building the thing that changes a thousand prices took weeks — which is weeks of chances to ask “wait, should it?” Now the agent can do it this afternoon, so the question never gets asked. The velocity at the front of the pipeline erases the pause that used to protect the back of it.
How to prevent it
Separate the goal from the authority. An agent can be trusted to propose long before it can be trusted to act. Start every autonomous workflow as a recommendation engine: it does all the analysis, drafts every change, and a human approves the batch. Promote it to acting on its own only for the specific actions where being wrong is genuinely cheap and the action is reversible. “Reversible and low-stakes” earns autonomy. “Irreversible and customer-facing” never does, no matter how good the demo was.
Scale autonomy to reversibility, not to confidence. The agent’s confidence is not evidence — it’s a token predictor’s self-report, and it reads exactly the same whether the agent is right or catastrophically wrong.
The serious team fix
Three things, reinforcing each other:
- A human approval gate on every irreversible or customer-facing action. The agent proposes; a person promotes. AI makes the proposal cheap and complete — the diff, the rationale, the expected impact — so the human is reviewing a decision, not redoing the work. The gate comes off only per-action, only after that action has earned it by being reversible and low-stakes, and the decision to remove it is itself a reviewed decision.
- Bounded objectives and a kill switch someone has actually pulled. “Propose changes to these 100 pages for approval” is a bounded objective. “Increase revenue” is a wish. The agent gets explicit limits — what it may touch, how much, how often — and a stop control that’s been exercised in a drill, not just wired up and assumed.
- Guardrails that don’t depend on a human watching. If the agent can act unattended, the protection has to be unattended too: automated checks that halt the loop when the target metric moves the wrong way, alerts that page a person, an audit trail of every action and its reason. The fix for “nobody’s watching at 2am” is not “watch at 2am” — it’s a system that stops itself.
The dream is an agent that runs the business. The reality, today, is an agent that’s an excellent analyst and an unaccountable operator. Let it analyze freely. Make it ask before it acts.