Five Things We Measured About AI-Assisted Coding (And What They Actually Mean)

There is no shortage of opinions about what AI is doing to software. There's a lot less measurement. So over the last stretch we kept doing the same boring thing: pick a claim everyone repeats about agents, MCP, or AI-written code, and check it against real public data — registries, transcripts, GitHub, package indexes — with no model in the loop, so every number is reproducible. We shipped a small tool for each. Here are the five findings that changed how we think, and the one that embarrassed our own hypothesis.

1. The official MCP registry is 84% hosted — and half of what's left won't boot

We pulled all 306 unique servers from the official MCP registry and measured the drop-off from "listed" to "actually runs." Two surprises. First, 84% of registered servers are remote-only — hosted HTTP endpoints, not packages you install. The mental model of "MCP server = a thing you npx locally" is already the minority. Second, of the 44 you can install, all 44 resolve (good — no dead pointers), but only 47% complete a handshake with no configuration. The rest need credentials or are abandoned, and the listing won't tell you which.

What it means: "in the registry" is not "runs." Pin the version and smoke-test the handshake before you wire a server into an agent. (deep dive →)

2. 43% of MCP tool parameters have no description

The model never sees your tool's code — it picks the tool and fills the arguments from the name, the description, and the parameter schema, and nothing else. So we connected to the eight official MCP reference servers and counted. Every tool has a description. But 43% of all parameters have none — and on the most-installed servers it's worse: git 79%, filesystem 72%. read_file's path argument is a bare string; the agent guesses what it means.

What it means: the documentation debt that makes agents misbehave lives at the parameter level, where "it's obvious" feels true to you and isn't to the model. Document parameters, give free-form strings an example or enum, before adding more tools. (deep dive →)

3. 70% of the package names an LLM would invent are already registered

LLMs hallucinate dependencies predictably — a USENIX 2025 study found 19.7% of AI-generated samples cite a non-existent package, and 43% of those names recur on every re-run. That predictability is the attack ("slopsquatting"). So we generated the "conflation" names the study describes (express-mongoose-style mashups of two real packages) and looked them up: 70% are already registered on npm, many with near-zero downloads.

What it means (stated honestly): registered ≠ malicious — most are benign abandoned packages. But the namespace LLMs hallucinate into is densely occupied by low-trust packages, so "it installed fine" proves nothing. Check an agent's new dependencies against the registry before you trust the import. (deep dive →)

4. Your repo's agent-instruction files have probably already drifted

Teams now keep CLAUDE.md, AGENTS.md, .cursorrules, and .github/copilot-instructions.md side by side, all supposedly saying the same thing. We checked real repos. browser-use/browser-use's CLAUDE.md and AGENTS.md are 1% similar — two files describing the same project differently, so a Claude session and a Codex session get materially different briefings. langchain keeps them byte-identical today, with no link between them — drift one commit away. The well-run repos (uv, airflow, prisma) use one canonical file and make the rest symlinks or one-line redirects.

What it means: pick one canonical instruction file (AGENTS.md is becoming the cross-tool standard) and make the others redirects. Don't hand-maintain N copies of your rules. (deep dive →)

5. The one that embarrassed our hypothesis: AI PRs don't dodge review — the repos do

Every 2026 report says agent PRs are bigger and get less review. We tried to confirm it the clean way — a PR is "agent" if its commits carry the Claude Code trailer — and measured 300 merged PRs across 18 agent-using repos. The hypothesis lost. Agent PRs were not bigger (median 39 lines vs 50; humans bigger at the tail). And the review gap basically vanished: 69% of agent PRs were rubber-stamped — and so were 66% of the human PRs. 79% of agent PRs and 67% of human PRs merged in under thirty seconds.

What it means: the agent didn't lower the review bar; the bar was already on the floor for everyone. The "AI skips review" panic, in the repos that adopt agents fastest, is a self-merge-culture problem, not an authorship one. (Caveat we put up front: that sample skews small/solo — run the measurement on your own repo, which is why we shipped it as a tool.) (deep dive →)

The thread running through all five

Write them out and a pattern appears. Every one of these is a place where the convenient assumption ("it's listed, so it works"; "it installed, so it's real"; "the name's obvious"; "the rules are in sync"; "the AI is the risky author") is exactly the thing that doesn't hold — and where nobody was measuring, because measuring is less fun than having an opinion.

The cheering take on AI coding and the doom take are both mostly vibes. What we keep finding, when we actually count, is more boring and more useful: the tooling is young, the defaults are optimistic, and a thirty-second deterministic check catches the gap between what a thing claims and what it does. None of these findings is a scandal. All of them are a reason to add a smoke test.

If there's one habit worth taking from a month of this, it's that: when an agent hands you a server, a dependency, a config, or a PR, the question isn't "is this fine" — it's "what's the cheap check that would tell me it isn't." Usually there is one. Usually nobody runs it. Each of these five is one we made runnable.

Every measurement here is reproducible — the tools and datasets are public. → Talk to us if you're putting real checks around AI-assisted development.

---

Five deterministic studies, each with its own write-up and tool on deemwar.com/insights: the MCP registry boot funnel, MCP tool-doc coverage, slopsquat occupancy, agent-instruction drift, and the agent-vs-human PR review measurement. Numbers were collected live in June 2026 and are exact; where a finding contradicts a popular narrative (or our own hypothesis), we led with that rather than buried it. Prior art — the USENIX hallucination study, the DORA/Cortex PR reports, NimbleBrain's registry analysis, Simon Willison's and Invariant's MCP work — credited in each deep dive.