The Autopilot With No Pilot
An AI agent has a business goal and production write access. Nobody defined the limits between them.
What it is
The Autopilot With No Pilot is an autonomous agent allowed to make consequential changes without a defined authority boundary, approval policy, or recovery plan. “Grow revenue” becomes permission to change prices. “Clear the queue” becomes permission to close customer cases. The objective is doing the work of an access policy.
How it happens
The demo is intoxicating. The agent reads the data, forms a plan, takes the action, and reports back. The leap that feels tiny and isn’t: pointing that same loop at the live system. The autonomy was the impressive part, so nobody wants to constrain it.
In an illustrative inventory workflow, an agent sees an old sales feed and decides that stock is not moving. It marks down a whole category. The API accepts every price update, so the run reports success. Restoring the prices later cannot undo orders already placed at the discount. The missing check was whether the data was fresh and the action authorized, not whether the API returned 200.
Warning signs
- The agent can change prices, send messages, or move money beyond an explicitly approved scope.
- A prompt says “ask first,” but the tool credentials can bypass that instruction.
- Nobody can distinguish a proposed change from one already applied.
- The only protection is a person noticing a dashboard at the right moment.
- The stop control has never been tested with queued or in-flight work.
Why it’s dangerous
A loop can repeat a mistaken decision across many records before anyone sees the result. Humans and ordinary scripts can do this too; an AI agent adds the ability to choose actions from an open-ended goal. Success at calling tools is not evidence that those choices were appropriate.
OWASP’s guidance on excessive agency identifies excessive functionality, permissions, and autonomy as causes of harm, and recommends approval for high-impact actions. The lesson is to constrain what can happen, rather than hope the agent’s explanation will reveal a mistake in time.
How to prevent it
Separate the goal from the authority. Let the agent prepare the actual proposed changes and supporting evidence before approval is needed. Put the approval check at the tool or service boundary, so a different prompt cannot skip it. Bind approval to the specific records and changes reviewed. If relevant state changes before execution — the price, available stock, or input freshness — recheck the preconditions and invalidate the proposal when they no longer hold. Approval of yesterday’s preview is not approval of whatever the tool happens to find today.
Grant unattended execution for defined actions whose consequences the team has evaluated. Set limits on records, amounts, frequency, and data freshness. Keep human review for consequential exceptions outside those bounds. For the price example, the execution service could enforce a minimum margin, a maximum batch size, and a fresh inventory snapshot. Those limits still apply when a person approves the proposal. Sending a thousand price changes to a reviewer with an “Approve all” button supplies a signature more readily than scrutiny.
The serious team fix
- Make authority enforceable. Give the agent only the tools and permissions needed for the approved workflow. Separate preparing a change from executing it when approval is required. Record the proposal, the authorization, and each attempted write. Give writes stable operation IDs or another deduplication mechanism where retries could repeat a consequential action.
- Test the stop and recovery paths. Confirm that stopping the workflow prevents new work and cancels or blocks queued actions. Identify writes already in flight; stopping the agent process does not necessarily cancel a request it sent. Rehearse recovery for the actions that can be reversed and escalation for those that cannot.
- Check outcomes as well as API responses. Reject stale inputs and out-of-bounds changes before execution. Monitor results, stop on defined breaches, and alert an owner. A falling business metric may arrive too late to be the first safeguard.
An agent can do the analysis, prepare the change, and carry out authorized work. “Nobody had to watch” is an achievement only when someone designed what happens while nobody is watching.