AI-native engineering · assessment + one-day session
Is your team's use of coding agents actually working?
Your team uses coding agents, and three questions usually have no answer yet: how much agent-written code reaches production without a human who understands it, whether the tests would fail if that code were wrong, and what the agents can read and run. The assessment answers them from your repos and CI. Then, on the day, we work alongside your engineers to set up the instruction files, guardrails and CI checks in your codebase.
Email io@deemwar.comSold per day. Rate on request.
Transcript
Most teams adopted coding agents one engineer at a time. Nobody can say how much agent-written code reaches production unreviewed, whether the tests would catch it being wrong, or where the secrets end up. The work has two parts. First, an assessment: we read your pull requests, agent instruction files, tests and CI, read-only, and score each area with the evidence behind it. Then one day in your codebase with your engineers. In the morning, the team writes the agent instruction files and sets which files, secrets and commands the agents can reach. After lunch, an engineer does a real backlog ticket with an agent while the team watches. In the afternoon, checks go into CI: secret scanning on agent output, a PR size limit, and spot checks where the code is broken on purpose and the tests must fail. The day ends with a retro, an owner for the setup, and three numbers to measure for thirty days. The files and checks live in your repos. To start, email io@deemwar.com with your team size and the agents you use.
1. Before the day: the assessment
A week before the session; about two hours of the team’s time.
A 15-minute questionnaire
For the lead and two or three engineers: which agents, for which work, what they are kept away from, what has already gone wrong.
Read-only access to one to three main repos and their CI
Pull request history, agent instruction files, test runs, pipeline config. Revoked afterwards; a screen share works instead.
A 30-minute call
With two or three engineers: who prompts, who reviews, what merges without a second look.
What the assessment looks at
Which agents, for which work, how much of each merged PR is agent-written, which tasks they are kept away from.
Repo-level agent instructions (AGENTS.md, CLAUDE.md, rules files, skills): present, current, followed?
How agent output gets reviewed: who reads it, what gets rubber-stamped, how big the PRs are.
Are tests written by the same agent that wrote the code? Do they fail when the code is wrong?
What the agents can read and run: env files, tokens, production credentials, permission modes, logs.
What can be seen of model and tool cost today. Usage-billed API keys show it per run; seat-based tools such as Copilot or Cursor do not.
A baseline from your PR history: cycle time, rework, escaped defects.
Each area gets a score from 1 to 5 with the evidence; the three highest-value changes become the plan for the day.
2. The day, hour by hour
On site or remote, with the engineers who use agents. Your codebase, your tickets. If a block runs long, the last CI check moves to the 30-day follow-up rather than being rushed.
- 09:00
Findings
The assessment scores and the evidence behind each one. The team corrects anything read wrong from the repos.
- 10:00
Set the house standard
Agent instruction files (AGENTS.md, CLAUDE.md) for one to three main repos, written with the team: conventions, build and test commands, files the agent must never touch. Unused skills and MCP servers removed; one added only if a real ticket needs it.
- 11:30
Guardrails
Permission modes, which files and secrets the agents can read, and which commands run without asking, configured for the agent tools in use. A cost guard per run where the provider exposes usage (API keys). Optional: a local check before an agent runs a command, for dangerous commands and prompt injection. Rules first, a small local decision model for the long tail, tuned on your own command history; never the model alone.
- 12:30
Lunch
Lunch break.
- 13:00
One real ticket, live
An engineer does a backlog ticket with an agent under the new standard while the team watches: how the work is split, what the prompt says, where the agent is stopped and a human takes over.
- 14:30
The verification loop
Checks wired into CI for your main repo on the day: secret scanning on agent output, a PR size limit, a review checklist for agent PRs, and spot checks on two or three critical paths that the tests fail when the code is broken on purpose. Other repos within the week.
- 16:00
Team session
A retro: what the team keeps and drops, who owns the agent setup from now on, and the three numbers to measure for 30 days.
What you keep
- A written report after the session: the score per area, the evidence, the three highest-value changes, and a 30/60/90-day plan with owners.
- Agent instruction files committed to your one to three main repos.
- A permission and secret guardrail configuration for the agent tools you use.
- CI checks: secret scanning on agent output, a PR size limit, and wrong-code spot checks on two or three critical paths.
- A review checklist for agent pull requests.
- A recording of the team session, using your meeting tool, if you want one.
- A call 30 days later to look at the numbers together.
- The trapCode merges that nobody understandsWhat we do instead
Small PRs, and a named reviewer who can explain every agent change
- The trapTests that agree with the bugWhat we do instead
Break the code on purpose; the tests must fail
- The trap"Done" that is not doneWhat we do instead
Evidence the work is live, not a report that it is
- The trapAgents with reach into credentials and productionWhat we do instead
Secrets kept out by configuration, passed by name at run time
- The trapCost surprisesWhat we do instead
A cost guard where usage is exposed, and a smaller model for simple decisions
- The trapMeasuring vibesWhat we do instead
A baseline of real numbers, then the same numbers 30 days later
The traps we see most
What goes wrong when teams adopt coding agents, and what we do instead. Several we have hit in our own agent work.
Code merges that nobody understands
Agent PRs are large and look plausible, so review turns into a skim. Generated code is still your code, but nobody on the team could explain it.
Small PRs by rule, a checklist written for agent output, and a named human reviewer who can explain every agent change.
Tests that agree with the bug
The same agent writes the code and the tests, so the tests encode its misunderstanding and pass.
Spot checks that break the code on purpose and confirm the tests fail, starting with your critical paths.
"Done" that is not done
The agent reports success and the pipeline is green, but the test result came from a cache, or the API said 201 while nothing was delivered. We wrote up six of these from our own systems.
Every task ends with evidence the work is live, not a report that it is.
Confident plans that are wrong
An agent writes a convincing plan and the team builds on it. The mistake shows up a week later, in code.
A second, independent review of the plan before any code: a different agent or person who did not write it, checking the claims.
Stacked AI reviewers nobody measures
Two or three AI review bots comment on every PR. Nobody knows which ones catch real bugs and which only add noise, so people stop reading all of them.
Count what each reviewer catches and what it gets wrong for a few weeks; keep the ones that earn their place.
Agents with reach into credentials and production
Agents read .env files, run commands with production credentials, and transcripts with tokens land in logs.
Agents kept away from secret files by configuration, secrets passed by name at run time, and secret scanning on agent output.
Rules sprawl nobody maintains
Everyone adds lines to the instruction files and nobody removes any. They contradict each other and the agent follows whichever it read last.
One owner, short files per repo, and a review whenever a mistake repeats.
Adoption by decree, no shared standard
Leadership asks for "real usage, not experiments", and each engineer invents their own way. Quality depends on who held the agent.
One house standard in the repo that the whole team set together, and the team in the room for the day, not one champion.
Non-engineers shipping through agents
Operations or sales staff ship changes with agents into systems that bill customers, through a review process built for engineers.
Clear paths for who may change what, with the same checks on every change whoever made it.
Cost surprises
Long sessions and the biggest model for every step multiply cost, and nobody sees it until the invoice.
A cost guard per run where your provider exposes usage, and a smaller model for simple decisions.
A pilot that never reaches production
A demo works on a laptop and stalls on review, security and ownership questions nobody planned for.
We start from a real ticket in the real codebase, so the first output goes through your normal pipeline.
Measuring vibes
"It feels faster" is the only metric, so nobody can tell whether to invest more or pull back.
A baseline of cycle time, rework and escaped defects from your PR history before the day, and the same numbers 30 days after.
Failures from our own systems
A cached test result
A Go test stayed green while the bug it guarded against was present, because the result came from cache. A test now only counts after a fresh run.
A 201 that delivered nothing
An API returned 201 Created and a media ID four times while the post went live without its image. Checks now read the published output, not the API response.
A command check we measured
In our own test, on 20 destructive shell commands our hand-written rules had never seen, the rules caught 1; a small local decision model caught 16, with zero false alarms on 20 safe look-alikes. So we run both: rules for what we know, the model for the long tail.
Agent work checked against its task
Our engineering uses coding agents, and every 30 minutes a separate process compares what each agent produced with what it was asked to do.
Questions
Who is it for?
Teams where several engineers already use Claude Code, Codex, Copilot, Cursor or similar and nobody can yet show how much of that code is reviewed and tested; teams whose agent use grew one engineer at a time; leads who must answer "is this working, and is it safe?" for a board or a customer audit.
Who is it not for?
Teams that want a strategy deck rather than changes in their code; teams that cannot give read access to code or share a screen; leaders who want the standard set without the engineers who use the agents.
What does the team provide?
A lead who can approve changes on the day; two or three engineers for the questionnaire and the 30-minute call; read-only access to one to three repos and their CI, or a screen share; one real backlog ticket; and the engineers for the full day.
Production access?
Not needed. Read-only access to code and CI, revoked afterwards, or a screen share.
Which agents?
Claude Code, Codex, Copilot, Cursor, or whatever the team already uses.
Remote?
Yes, with cameras on and the team free for the day.
Cost per task?
Only where the provider bills by usage, such as API keys. Copilot and Cursor seats do not expose it, and the report says so.
Price?
Per day, in writing in the first reply.
A product to buy?
No. This is a service: the charge is for people’s time (the assessment and the session).
Next step
Email io@deemwar.com with your team size, the agents you use, and the one thing about them that worries you most. The reply has a proposed assessment week, a session date and a written rate.
io@deemwar.comOr send it from here: