Artificial Analysis Decision Index v2 ranks top decision models
The heavier private-dataset weighting and confidence-aware scoring make it harder for models to game public benchmarks, shifting how leaderboard positions are earned. Teams selecting a decision model should weigh the 60x cost spread — Celeris-1.1 Decision sits on the Pareto frontier at US$14 per 1M tasks versus Drex Max v1.1 at US$102 — since token price alone understates real task costs.
Artificial Analysis released Decision Index v2, an updated evaluation for decision-making models with a new Decision Score that rewards calibrated confidence alongside correctness. Private datasets now make up 45% of the Index, up from 25% in v1, including the new AA-Paperclip benchmark of 1,500 documents. Drex Max v1.1 from Nace AI leads at 73, within 1 point of GLiDE v1.2 from Fastino Labs and Mammoth 27B from Base Labs at 72. Cost per 1M Tasks varies by more than 60x, from US$14 for Celeris-1.1 Decision to US$859 for Clef.
Introducing the Artificial Analysis Decision Index v2: a new proprietary evaluation, updated benchmarks, and a headline metric that rewards model calibration
We’ve released the second version of the Artificial Analysis Decision Index to better reflect real-world utility and guard against benchmark gaming. 45% of the Index is now built on private datasets, up from 25% in v1. Drex Max v1.1 from @NaceAI narrowly leads among decision models at 73, within 1 point of GLiDE v1.2 (@fastinoAI) and Mammoth 27B (@baselabs)
Updates in Decision Index v2:
➤ Decision Score: We adopt a new headline metric that rewards a model for both correctness and calibrated confidence. Each question scores from 0 to 100: right answers score above 50, wrong answers below 50, and confidence in its top answer moves the score within each half
➤ AA-Paperclip (30%): We added a new private benchmark of 1,500 documents including resumes, receipts, invoices, and AI and machine learning research papers. Models make decisions and extract facts from each document and are scored jointly as a single decision, correct only if all 15 answers are right
➤ AA-Symmetry (15%): This evaluates whether models change their answers for equivalent questions worded differently. We test 120 questions with variations such as reordered options, swapped labels, and negated yes/no questions, then measure how consistent the model’s probabilities are across those variations
➤ AA-Omniscience-Decision v1.1 and HLE-MultiChoice v1.1 (15% each, down from 25%): AA-Omniscience-Decision now varies the wording of its “I don’t know” option across ten alternatives. HLE-MultiChoice adds an option to decline using the same ten phrasings
We’ve also added EvalSafe-ONET-AA (15%), our implementation of @typesafeai’s ONET benchmark. We have retained When2Call from @NVIDIAAI in the Index at a decreased weighting of 10% (down from 25%), and removed WildGuardTest given its high saturation
Key results:
➤ Three decision models sit within 1 point at the top of the Index: Drex Max v1.1 from Nace AI and GLiDE v1.2 from Fastino Labs both score 73, with Mammoth 27B from Base Labs scoring 72
➤ Cost per 1M Tasks varies by more than 60x across decision models: Celeris-1.1 Decision ($14), Decider v1.2 ($68) and Drex Max v1.1 ($102) sit on the Decision Index vs. Cost Pareto frontier, while Clef costs $859. Token price is not the only cost driver: most decision models, including Drex Max v1.1 and Decider v1.2, bill a document’s input once for all its questions, while GLiDE v1.2 and Decider Flash v1.2 bill it for every question. This means GLiDE v1.2 costs $780, nearly 8x higher than Drex Max v1.1 despite only 33% higher list prices
➤ Language models can score higher, alongside higher Cost and Time per Task: Claude Haiku 5.5 (Max) scores 77 and GPT-6 Luna (Max) scores 76 on Decision Index v2, putting them ~3 points ahead of Drex Max v1.1. GPT-6 Luna (Max) costs $1,530 per 1M Tasks vs. $102 for Drex Max v1.1 (15x the cost) and takes 11.34 seconds per task vs. 275 milliseconds (~40x slower)
