Princeton and Google paper: 4B decision model cuts agent costs
Confidence thresholds fitted on your own agent's traces, not generic accuracy, are what decide whether a small router model helps or hurts. Practitioners building tool-using agents can cut costs by training a small decision model on trajectories their own agent actually reaches, mistakes included. Compared with routing every tool call blindly, the thresholded approach preserved success within 3 points.
A Princeton University and Google paper (arXiv 2610.21412) finds that the decision model most accurate on isolated questions solved only 69.5% of agent tasks, while a 4B model trained on agent states solved 91.5%. Their harness routes multiple-choice agent decisions, such as which tool to call next, to a fast decision model, accepting answers only above a confidence threshold fitted on the agent's own runs. They trained AgentDecision-4B on states the agent actually reaches, including states after its own mistakes. On tasks resembling the training data, this made agents 81.1% to 89.0% cheaper and up to 2.5x faster, with success staying within 3 points. Handing every tool choice to the small model with no threshold lowered success by 19.0 points.
New Princeton Univ + Google paper finds that the decision model most accurate on isolated questions solved only 69.5% of agent tasks, while a 4B model trained on agent states solved 91.5%.
Agent harnesses call a big LLM at every step, which makes long tasks slow and costly. Yet most of those calls just pick from a list, like which tool to use next.
Their harness asks a fast decision model each multiple-choice question and uses its answer only above a confidence threshold fitted on the agent's own runs. They also trained AgentDecision-4B on states the agent actually reaches, including after its own mistakes.
On tasks like their training data, 4B models trained this way made agents 81.1% to 89.0% cheaper and up to 2.5x faster. Success stayed within 3 points, while handing over every tool choice with no threshold lowered success by 19.0 points.
To hand routine agent choices to a small model, train it and set its confidence threshold on your agent's own runs.
– arxiv. org/abs/2610.21412
Title: "JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs"
