The Fake Confidence Refactor
The rewrite looks cleaner. The claim that it behaves the same still needs evidence.
What it is
The Fake Confidence Refactor is an AI-generated cleanup accepted as behavior-preserving without checking the behavior that matters. The diff looks tidy, the description sounds certain, and an untested edge changes. The refactor is a regression with a great-looking PR description.
Preserving observable behavior is part of Martin Fowler’s definition of refactoring. A deliberate behavior change can be valuable, but it needs its own requirement and review instead of riding along under “cleanup.”
How it happens
Someone asks AI to “make this more idiomatic.” It merges similar functions, removes apparent duplication, and simplifies conditions. Those may be useful changes. The trouble starts when similarity is treated as proof that two paths have the same contract.
An illustrative JavaScript rewrite replaces
limit ?? 100 with limit || 100. Both use 100
when limit is missing, but only the second also replaces
zero. If zero means “return no records” in this API, the
cleanup changes behavior. MDN’s
operator reference explains this distinction. A test using only undefined and 10 would not catch it.
This is a small example of a larger review problem: a structural
rewrite can smuggle in semantic changes that are easy to spot alone
and hard to spot among hundreds of moved lines.
Constraints may live in call sites, configuration, old bug reports, or downstream consumers. AI can investigate those sources when the task includes them. A broad rewrite of one file does not establish that all those contracts survived.
Why it’s dangerous
A changed edge may look unrelated to the cleanup when it fails later. The reviewer saw improved structure; the on-call engineer sees a wrong response under an unusual input. A large diff makes the connection harder to trace, especially if it mixed restructuring with changes in behavior.
AI reduces the effort of producing a broad rewrite. That does not reduce the evidence needed to trust it. Passing tests are useful evidence for the cases they exercise; they cannot settle a contract they never checked. The problem is overstating what the green result establishes.
How to prevent it
State what must remain unchanged, then choose checks that could expose a plausible regression. For the limit example, include zero, a positive value, and omitted input, with expected results tied to the API contract. Where the existing behavior is poorly documented, characterization tests can record it before the rewrite. Resolve suspected bugs separately rather than blessing every old result as correct.
Match effort to stakes. Renaming a private helper needs less evidence than reorganizing payment handling. Separate mechanical moves and renames from changes to conditions, defaults, or execution order. An editor’s rename operation may be the right tool for a mechanical edit; an unconstrained model rewrite adds little there. For a larger restructuring, AI can draft comparison tools and tests, then use their results to investigate differences.
The serious team fix
- Name the preservation claim. A review should say which behavior is unchanged and what checked it: response fields, authorization, ordering, rounding, or another concrete contract. Name important gaps rather than calling the whole rewrite “equivalent.”
- Compare old and new where it matters. Use representative inputs and compare relevant outputs and side effects. For an event producer, a matching return value is insufficient if the new path publishes twice. Decide which differences are permissible before interpreting the results. Helper names may be incidental; event ordering or a log format consumed by an alert may be contractual. A textual diff alone cannot decide that for the team.
- Isolate writes during comparison. Replay stateful paths in a sandbox with controlled starting state. Shadowing live read-only traffic can help where it is safe, but running both versions must not charge a customer twice, send duplicate messages, or mutate shared data. Control clocks and IDs where possible. If you normalize them during comparison, check that relationships still hold — for example, that the event references the order actually created. Document what the comparison cannot establish, including timing or concurrency behavior absent from the replay.
Before merging, the reviewer should be able to name the structural improvement and the evidence covering its main regression risks. “Cleaner” describes a preference. It does not explain why this particular rewrite was worth doing.