Coverage Theater
AI-generated tests can agree with the code and still miss the requirement.
What it is
Coverage theater is a test suite that measures AI-generated code against the same model’s reading of that code, rather than against the required behavior. The coverage number climbs. The suite is green. The implementation has only been mirrored.
How it happens
Write the function with AI. Then: “now write tests for
this.” The AI reads the handler it just wrote, walks the
branches, and asserts whatever the implementation happens to
return. If the handler returns HTTP 200 with a null
body for a missing record, the test asserts that response
— even if the API contract requires a 404. It copies
the observed behavior because the model was shown the handler,
not the contract. The PR shows a large test addition and a green
coverage report. The reviewer thinks: tests, great.
Nobody zoomed out to ask: what requirement do these tests prove? The AI never had the requirement. It had the code. It wrote tests about the code. The tests are a mirror, not a check.
Why it’s dangerous
A mirrored suite can still catch regressions, but its green result may be mistaken for evidence that the original behavior was right. Bugs in the implementation can be transcribed into assertions — the off-by-one becomes the expected value, the swallowed exception becomes the expected silence. Tests coupled to internal details can also fail during a behavior-preserving refactor, adding noise when the team needs a reliable signal.
And it compounds. The next AI session inherits the high coverage number, treats the suite as a safety net, and generates new mirrored tests against new mirrored code. The number keeps climbing without establishing that the product requirement is met. Engineers have always been able to write tautological tests; AI generation can scale this failure and make it look polished. A study of generated test oracles observed tests reproducing actual rather than expected behavior, while also finding that generated tests can detect faults. The risk is misplaced confidence, not the use of AI to write tests.
How to prevent it
For important behavior, expected results need grounding beyond the current implementation. The source can be a written spec, a ticket, an API contract, example inputs and outputs from the real world, the prior version’s behavior — anything that states what the system should do. Code-informed tests can still catch regressions and edge cases; they cannot establish an unstated product requirement by themselves.
Match rigor to stakes. A typo fix doesn’t need a new test. A new branch in business logic often does. Give AI the requirement and ask it to draft a test against the requirement. Then, before you trust the green, try a plausible faulty change and rerun the suite. If the targeted test still passes, investigate the assertion or the test setup. The friction signal: when writing a test feels like rewriting the function in the negative, you’re mirroring.
The serious team fix
Three things, reinforcing each other:
- A team habit of writing tests from intent, not implementation. For any change with meaningful behavior, the author has to be able to answer one question: what requirement does this test prove? “It covers the function” is not an answer. The requirement might be a ticket, a contract, an example, a prior version’s output — but it has to exist outside the code under test. PR reviews ask the question out loud.
- An AI-assisted adversarial test pass. Give AI the spec, contract, or example data and ask it to draft expected outcomes and edge cases. Compare these with the implementation and existing tests. A mutation testing pass can introduce small code changes (such as negating a comparison) and report which ones survive. Investigate survivors; some are equivalent changes or outside the behavior the suite is meant to cover.
- Integration checks and targeted mutation testing. Keep fast unit tests, including mocks where the contract is clear. Add tests against a real database, queue, or sandboxed API for boundaries where mock assumptions might be wrong. Use a mutation testing tool on important logic when the cost is justified. Coverage shows which code ran; surviving mutations point to assertions worth examining.
Coverage helps reveal untested code. Confidence in correctness also needs expected outcomes tied to requirements and boundary checks that can expose wrong assumptions.