The It-Worked-Once Deployment
The service stayed up for thirty seconds. Operations would like to discuss the remaining hours.
What it is
The It-Worked-Once Deployment treats a successful local run as enough evidence to release a service. The team has seen the happy path but has not checked the production configuration, likely failure modes, or recovery procedure. A green health endpoint becomes the opening ceremony for an unplanned on-call shift.
The condition the demo never exercised
In an illustrative Amazon SQS worker, processing a message locally
succeeds with a developer’s credentials. The deployed role allows
sqs:ReceiveMessage but omits sqs:DeleteMessage,
which are separate
permissions. The worker performs the business action, logs the failed
deletion, and continues polling. After the visibility
timeout, the undeleted message becomes available again. The process
remains healthy while the action can repeat.
A startup check would miss this. A release check that sends a test message through the deployed worker, verifies its effect, and confirms acknowledgment would expose it. Use a test identity configured like the deployed role, not the developer’s admin profile. Duplicate delivery still needs handling even after the permission is fixed; the visibility timeout is not an exactly-once guarantee.
Why it’s dangerous
The agent’s completion report may accurately say it ran the worker and processed a message. The team reads “tests pass” as release readiness without noticing which credentials and artifact were exercised. That leap turns an honest local result into an unsupported deployment claim. The first check of the deployed identity then happens on a customer’s order.
AI can help build deployment scripts, realistic checks, and operational documentation as well as application code. Those artifacts still need to fit the system and be exercised. Generating twenty alert rules is easy; deciding which failure should wake someone, and maintaining those rules as the service changes, is continuing engineering work.
How to prevent it
Name the release risks introduced by the change. For the worker, check the deployed role, acknowledgment, retries, concurrent messages, and shutdown with work in flight. Use isolated test records and a staging environment that matches those properties. It need not duplicate every production resource or its full capacity to answer a bounded question.
Deploy an identifiable artifact through a repeatable procedure. Observe the changed behavior before expanding exposure. Google’s SRE guidance on canary releases explains why the population, duration, and metrics need to represent the risk. A quiet canary that receives no relevant work has not supplied reassuring evidence.
The serious team fix
- Make release evidence specific. Record the artifact, environment, checks, and known gaps. For the worker, watch completion and failure counts, message age, and repeated processing as well as process health. In this example, record processing and acknowledgment as separate outcomes: an increasing “processed” counter can conceal repeated work. Alert on failed acknowledgments and rising message age, with an owner who can pause consumption.
- Decide how to stop harm. Rehearse pausing new work and accounting for in-flight requests. Reverting the binary cannot unsend an email or repair data already written. Schema changes may prevent an older binary from running; keep versions compatible during rollout or prepare a tested forward-repair procedure. Name who can authorize recovery and what evidence they need.
- Keep the release path affordable to maintain. Reuse an existing pipeline and operational conventions. A static copy edit needs less machinery than a billing worker. Give the service an owner and a concise runbook that has actually been used in a rehearsal; a generated recovery command nobody has run is another demo.