You don't need a frontier model for every decision in your agent loop

ignored h1

A project called Jeff landed on Hacker News this week: open fine-tunes of small open-weight models (Qwen3.5 and Gemma 4, at 0.8B and 2B parameters) that answer the same request format as openjevx — describe a situation, list the options, get back a calibrated probability per option from a single forward pass. No generated text, no parsing, no multi-second round trip to a frontier API. The whole thing was trained on one workstation GPU with synthetic data written by an open model, and the authors are upfront that it won't out-reason a large model on hard judgment calls — it's built for the much larger set of decisions that don't need one.

That distinction is the part worth pulling out, because it maps directly onto a mistake we see in a lot of agent code: every decision point routes through the same big model, because that's the model that's already wired up.

Most of what an agent loop decides is classification

Route this support message to the right queue. Is this file relevant to the task. Should this retry happen or should we give up. Which of these five near-duplicate search results is the one the user meant. Is the customer angry. None of these need a paragraph of reasoning — they need a probability distribution over a small, known set of options, returned fast and cheap enough that you can call it on every single step of a loop without thinking about the cost.

Sending that kind of question to a frontier model works. It also means every "obvious" branch in an agent's control flow pays the latency and cost of the same model you'd use to draft an email or write a function. At any real volume — an agent looping over hundreds of candidates, or triaging a queue continuously — that adds up in ways that change what you're willing to let the agent do autonomously.

What we actually run

We've been building on this pattern ourselves, not as a research exercise but as day-to-day infrastructure. jevx, one of our own internal skills, exists specifically so that classify/filter/triage/rank calls — "is this urgent", "which of these should I act on first", "would this pass review" — go to a small, fast decision model instead of being reasoned out inline by whichever large model is driving the agent that turn. We run our own decision model trained on our own machine, and we default to it for exactly the judgment calls Jeff is aimed at: yes/no gates, picking one of a few options, scoring a short list.

Training these ourselves has mostly been a story about where the compute comes from rather than how big the model needs to be — free-tier GPU hours are enough to fine-tune something in this size class in a single session, run as a batch job rather than an interactive one. The model doesn't need to be smart in the way a coding agent needs to be smart. It needs to be consistent, fast, and cheap enough that calling it isn't a decision in itself.

The practical effect on an agent loop is less about any single call being faster, and more about what becomes possible when a decision is cheap: you stop being stingy about how often the agent checks itself. A gate you'd only run once because it's expensive becomes a gate you run on every candidate, because it isn't.

The tradeoff is real, and worth stating plainly

A 0.8B model is not going to catch a subtle bug, resolve an ambiguous requirement, or write anything you'd want to ship unreviewed. Jeff's own documentation says as much — it approaches or beats larger judgment models on benchmarks, but "at this size their reasoning won't match [a much larger model's]." That's the right way to think about it: this is a tier below your main model, not a replacement for it. The skill is knowing which of your agent's decisions belong in which tier, and building the loop so the cheap tier is the default, not the exception.

If you're building agents that call a model on every step and haven't drawn that line yet, it's worth an afternoon: pull the classification-shaped decisions out of your loop, and see how many of them were never generation problems in the first place.