Stop Reviewing AI Code Harder. Start Checking It Cheaper.

We've argued elsewhere that the AI-code "verification gap" isn't a trust problem, it's a measurement problem: developers don't trust AI-written code, but they merge it anyway, because reviewing it is expensive and checking it usually isn't built. A review is a tired human forming a judgment about a diff. A check is a machine returning a verdict in milliseconds. The failure modes that bite hardest with AI code — the "looks correct but isn't" ones — are exactly the ones a human skim misses and a deterministic check catches every time.

This is the practical companion to that argument: the catalog. Below is every AI-code failure mode we've actually measured this year, paired with the thirty-second deterministic check that closes it. None of these requires trusting the model more or reviewing harder. Each one moves a worry off the reviewer's plate and into CI, forever.

The catalog

1. Hallucinated dependency. The agent confidently imports a package that doesn't exist — 19.7% of AI-generated samples do — and the name it invents is often already registered by a squatter (we found 70% of the obvious ones are taken). → Check: look up every new dependency against the live npm/PyPI registry. Non-existent, or one edit from a popular name, or near-zero downloads → flag. A registry call, not a judgment.

2. The lethal trifecta. Your agent's granted tools, taken together, can read private data and ingest untrusted content and exfiltrate — the prompt-injection data-theft precondition. No single grant is wrong; the union is. → Check: classify the whole grant set (.mcp.json + settings.json) into the three legs and fail if all three are reachable. Break one leg (the "rule of two").

3. Dead or undocumented MCP servers. You wire in a server from the registry assuming it works. 84% of the official registry is remote-only; under half of the installable servers boot clean, and 43% of tool parameters have no description, so the agent fills arguments by guessing. → Check: before depending on a server, complete an initialize+tools/list handshake and count undocumented parameters. "Listed" is not "runs"; "exists" is not "documented."

4. A committed secret in agent config. The agent (or a teammate) pastes a real token into .mcp.json or settings.json and commits it. → Check: scan the config for high-entropy literals in secret-named keys (and don't flag ${ENV_VAR} references). A regex, not a review.

5. Invisible / hidden instructions. A pasted file or tool description carries Unicode tag characters or a buried "ignore previous instructions" that the model reads and you don't. → Check: scan inbound text and tool descriptions for invisible Unicode and instruction-shaped strings. The model can't tell whose instruction it is; you can.

6. Instruction-file drift. Your repo keeps CLAUDE.md, AGENTS.md, .cursorrules — all supposedly saying the same thing — and they've silently diverged (one real repo's two files were 1% similar), so different agents follow different rules. → Check: diff the instruction files; if two substantive ones disagree, fail. Keep one canonical file and make the rest redirects.

7. Context burned on redundant work. The agent re-reads a file it just edited (the harness already confirmed the edit), re-runs the same command, re-greps the same pattern — ~85k wasted tokens in one real session — and the cost of an early big read compounds every turn it stays resident. → Check: scan the session transcript for duplicate reads and read-after-edit, and weight results by residence time. A transcript pass, not a code review.

8. The unreviewed merge. The scariest one, because it's the gate itself. Surveys say 71% won't merge AI code without review; merge logs say 79% of agent PRs merge in under thirty seconds with no review. → Check: measure your own repo's time-to-merge and review-comment count, split by author. If everything self-merges in a minute, your problem is a missing gate, not the AI.

How to actually use this

You don't need all eight on day one. The move is to pick the one failure mode you most worry about and ask the catalog's question: is there a deterministic test for it? Almost always there is, because these failures are structural — existence, documentation, drift, resource use, an open gate — and structure is exactly what machines check reliably and humans skimming a diff miss. Put that one check in a pre-commit hook or a CI step. Now it's closed for everyone, on every PR, without anyone forming a judgment.

Three properties make these worth more than they look:

• They're cheap to run and cheap to keep. A registry lookup, a regex, a handshake, a diff. Milliseconds, no model, no maintenance burden that competes with shipping.
• They never get tired. The reason review fails on AI code isn't that humans are bad at it — it's that the volume went up and the failures look fine. A check is immune to looking fine and immune to the 4 p.m. skim.
• They free human judgment for where it's irreplaceable. Every failure mode you make deterministic is one your reviewers no longer spend attention on — so the attention they do have goes to architecture, intent, and tradeoffs, which no grep will ever judge.

The uncomfortable truth under all of this is that "we review our AI code carefully" is, for most teams, a policy their merge log doesn't support. That's not a character flaw; it's an economics problem, and you fix economics by lowering a cost, not by exhorting people to pay it. Lower the cost of checking until skipping it makes no sense. Each row above is one we made cheap enough to keep.

We build these checks — and we measured every failure mode above before we wrote the check for it. → Talk to us if you want the gate around your agent-written code to be one your merge log actually backs up.

---

Every failure mode and number in this catalog links to its own measured write-up on deemwar.com/insights — registry hallucination occupancy, the lethal trifecta, the MCP registry boot funnel and tool-doc coverage, instruction-file drift, session waste and context carry-cost, and the agent-vs-human review measurement. This post is the index and the checklist; the deep dives are the evidence. All checks are deterministic — no model in the loop.