System1-Arena benchmark tests which decisions LLMs can make without reasoning
System1-Arena: Which Decisions Can Be Left to System 1?
The benchmark gives a concrete cost-accuracy-latency trade-off table for choosing between cheap probability-returning models and expensive reasoning models in production decision pipelines. Teams building flagging, escalation or agent control systems can use the 105-task and 138-environment breakdowns to match model class to task type. The finding that open LLMs beat System-1 models in closed loops challenges assumptions about small fast models in agentic settings.
A new benchmark called System1-Arena evaluates whether System-1 models, such as Jev and the OpenAI Decisions API, can handle discrete decisions without generative reasoning. It covers 92k questions across 105 tasks from 268 public sources, plus 138 closed-loop environments including web shopping, SWE-bench coding control, robotics, games and driving. The authors evaluated 30 models: 7 closed LLMs, 13 open LLMs and 10 System-1 models. The largest System-1 models matched or exceeded fast closed LLMs on non-agentic accuracy with better-calibrated probabilities at one-quarter of the latency, while reasoning LLMs were most accurate at about 70x the cost of a fast LLM. In closed-loop tasks, large open LLMs succeeded most often at up to 32.9%, while small System-1 models rarely succeeded.
System1-Arena: Which Decisions Can Be Left to System 1?
Large language models (LLMs) often make discrete decisions, such as flagging phishing or escalating to a human. System-1 models, such as Jev and the OpenAI Decisions API, return probabilities over options in a single forward pass, while System-2 approaches generate reasoning steps. Which decisions can System-1 models handle, and which require generative reasoning? We introduce System1-Arena, a benchmark that gives all models the same decision requests and scores their probabilities. It contains 92k questions across 105 tasks from 268 public sources, in four groups: text classification and knowledge, applied decisions, evaluation and control, and agent and action. It also includes 138 closed-loop environments, from web shopping and SWE-bench coding control to robotics, games, and driving; in real-time mode, the world evolves while a model decides. We evaluate 30 models: 7 closed LLMs, 13 open LLMs read at the answer token, and 10 System-1 models. The largest System-1 models match or exceed fast closed LLMs in non-agentic accuracy, with better-calibrated probabilities at one-quarter of the latency. A reasoning LLM is the most accurate, at about 70x the cost of a fast LLM. In closed loops, large open LLMs succeed most often (up to 32.9%); small System-1 models rarely succeed.
