Back to Insights
5 Aug 20264 min readAI Agents

Your agent didn't change. The cache did.

Someone ran a fifty-agent pipeline with real cost controls on it: a 200K token ceiling per agent, hard abort checkpoints, runaway guards, a per-agent cost ledger. Thirty-six agents finished 72% of the work on 3–4% of the weekly quota. The controls had already proven themselves across multiple production runs.

Then they hit the weekly limit, waited a few hours, and resumed the same pipeline. The remaining fourteen agents burned roughly 20% of the quota in about four hours — more than ten times the expected rate.

Every agent still reported under its 200K limit. No threshold tripped. No guard fired. No anomaly appeared in the ledger. The only signal was the account meter, after the money was gone.

Before the pause, a context prefix is a cheap cache READ at ~0.1x input price. After a pause longer than the cache TTL, the same prefix is re-ingested as a cache WRITE at ~1.25x. The budget guard counted only output tokens — 1.3% of the tokens in flight — so it never fired.

The pause is the cause

Prompt caches have a short time-to-live — minutes by default, an hour on the extended option. A wait of "fewer than six hours" expires all of them.

Before the pause, each agent's stable prefix was a cache read, billed at a small fraction of base input. After it, that identical prefix has to be re-ingested as a cache write, priced above base input, and then read again. Same workflow, same context, same code path. The cost per agent steps up by roughly the read-to-write ratio.

There is a perverse interaction hiding in that sentence, and it is the part worth carrying away: a pipeline that pauses because it hit a quota will resume into its most expensive possible configuration. The limiter and the cache work against each other. The safety mechanism sets up the expensive run.

Why nothing fired

Their post-mortem reported roughly 11.3M output tokens, 50.6M cache-write, and 819.5M cache-read.

Output was about 1.3% of the tokens that actually moved. The per-agent ceiling was measuring that 1.3%. It was never going to fire, no matter how far the run overran.

Nothing was broken about the guard. It was watching the wrong quantity. cache_creation and cache_read are separate token classes from output, so the entire class that changed was invisible to a ledger that summed output alone. No threshold crossed, no anomaly, no alert — right up until the account-level meter caught up days later.

The general shape

We keep meeting this same failure wearing different clothes.

A separate report the same week: an agent ran text-to-speech generation scripts on a developer's machine and produced about $19 of cloud spend on audio the application structurally could not play. The developer's own dashboard reported the correct figure for the month — because it metered the application's API routes, and the scripts were not the application. Their words for it were exactly right: invisible by construction.

Better still, that incident had a fix applied mid-day. 81% of the money was spent after the fix, because the fix landed on one script and not on its duplicate.

Three different incidents, one structure:

  • A budget control that runs inside the thing being budgeted can only count what that thing chooses to expose.
  • A meter keyed on a code path is blind to every other caller of the same credential.
  • A corrected code path can be bypassed by a copy of that code path.

The number that actually gets billed lives at the credential, at the API boundary — not at the call site, and not in the tool's own accounting of itself.

What to do about it on Monday

  • Track all four token classes separately — input_tokens, output_tokens, cache_creation_input_tokens, cache_read_input_tokens. Every API response reports them in its usage block. The failure mode is summing the wrong subset and calling it spend.
  • Treat any resume-after-idle as a cache-cold run. Expect the first pass over each context to be charged at write rates, and size the budget for that, not for the warm case you measured.
  • Meter at the credential, not the call site. If two things hold the same key, one meter that only knows about one of them is not a meter.
  • Know which of your controls can refuse. Most cloud "budgets" are notification-only — they will tell you, on a lag, that money is gone. A quota or a hard cap is a different object: it can say no. Only one of those two is a control.

None of this requires believing anything about how expensive agents are in general. It requires knowing that the quantity you are counting is the quantity you are billed for. Usually it isn't.

AI agentscost governanceFinOpsprompt cachingobservability

Found this useful?