Taste-Bench from Microsoft tests agent decision quality
The benchmark targets a capability gap that reasoning-token budgets do not close: knowing which branch to take before the evidence arrives. Teams building long-running coding or research agents can use Taste-Bench-style forks to evaluate direction-finding rather than only final task success, and the distillation result suggests a practical way to train that judgment into smaller models. It complements SWE-bench Pro instead of duplicating it.
Microsoft researchers and colleagues released Taste-Bench, a benchmark measuring how often frontier agents choose the better direction at decision points in long tasks.
The best model answers 59.7% correctly.
Forks are mined automatically from parallel attempts and detours in engineering and research runs, and forks with deciding evidence appearing later in the trajectory are harder, with a larger reasoning budget not raising accuracy.
The team distilled a teacher's outcome judgment into a student model, improving end-to-end success on held-out SWE-bench Pro tasks.
Banger paper from Microsoft and colleagues. We talk about human taste being important in this AI era. But for recursive self-improvement, an agent's taste also matters. The big question is: How often do frontier agents pick the better direction at a decision point in a long task? It's apparently just under 60% of the time, according to Taste-Bench. In this benchmark, each question shows a point in a trajectory where several directions are open, and one leads to a better outcome. The forks are mined automatically from parallel attempts and detours in engineering and research runs. The best model answers 59.7% correctly. Forks whose deciding evidence appears later in the trajectory are much harder, and a larger reasoning budget does not raise accuracy. The authors then distill a teacher's judgment of the outcome into a student model, which improves end-to-end success on held-out SWE-bench Pro tasks. Paper: arxiv.org/abs/2609.25804 Chat with Paper: academy.dair.ai/papers/the-tas…📊 5 ⚡
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Taste-Bench, Microsoft, agents, SWE-bench Pro.
Source attribution: curated from elvis (@omarsar0).
