System1-Arena benchmark tests which decisions LLMs can make without reasoning
The benchmark gives a concrete cost-accuracy-latency trade-off table for choosing between cheap probability-returning models and expensive reasoning models in production decision pipelines. Teams building flagging, escalation or agent control systems can use the 105-task and 138-environment breakdowns to match model class to task type. The finding that open LLMs beat System-1 models in closed loops challenges assumptions about small fast models in agentic settings.