Artificial Analysis Makes Time per Task Its Headline Benchmark Metric
Benchmarking by task completion time rather than tokens per second changes how verbose models rank: GPT-6 Astra beats faster rivals because it emits about 50k output tokens per task versus 120k–170k for competitors. Default-effort data shows medium effort cuts task time 3–5x at a modest intelligence cost, which directly informs cost-sensitive API choices. Developers comparing models for agentic workloads should re-read leaderboards with this metric in mind.
Artificial Analysis has updated its API performance benchmarking across roughly 1,000 serverless endpoints it tests multiple times daily. Time per Task, which divides a model's output tokens per Intelligence Index task by its End-to-End Output Speed, replaces raw streaming speed as the headline metric. End-to-End Output Speed now measures from request to final token, uses provider API-reported token counts instead of the o200k_base tokenizer, and every measurement now includes multi-turn agentic workloads across coding and knowledge work. The benchmark also tracks models at each lab's API default effort setting alongside maximum effort. OpenAI's GPT-6 Astra and GPT-6.1 Sol hold the Intelligence vs. Time per Task Pareto frontier from about 45 to 55 on the Intelligence Index, with GPT-6 Astra finishing tasks in roughly 17 minutes at max effort despite running at only 51 tokens/s.
Announcing the next version of Artificial Analysis API Performance Benchmarking: Time per Task is now our headline performance metric, measuring how long a model takes to complete a task rather than how fast it streams
The update applies to the ~1,000 serverless endpoints we benchmark multiple times a day for speed, latency and caching.
Key changes: ➤ Time per Task is now the headline metric to compare models: as workloads become more agentic, the time to complete a task matters more than speed per token. A model that is fast per token but verbose can take longer than a slower model that uses fewer tokens. Time per Task divides a model's output tokens per Intelligence Index task by its End-to-End Output Speed.
➤ End-to-End Output Speed covers the full request: speed is now measured from sending the request to receiving the final token. That includes the wait before the first token (prefill, and reasoning for models that hide it) as well as generation. Output tokens now come from each provider's API-reported counts instead of the o200k_base tokenizer, and we monitor for discrepancies in reported counts.
➤ Agentic workloads are now part of every measurement: we benchmark short, document-length and long-context prompts, plus multi-turn sessions across coding and knowledge work. Headline figures are the average across these workloads over the past 5 days.
➤ API default effort is now tracked alongside highest effort: most developers run models at the effort level each lab sets as its API default, so we now benchmark key models at that setting as well as at their highest effort. Effort names are not comparable across labs, so we use each lab's own default rather than one shared setting.
Key takeaways: ➤ OpenAI models make up most of the Intelligence vs. Time per Task Pareto frontier despite slower output speeds: across effort levels, GPT-6 Astra and GPT-6.1 Sol hold the frontier from ~45 to ~55 on the Intelligence Index. At max effort, GPT-6 Astra runs at 51 tokens/s but finishes a task in ~17 minutes, against ~26 for Claude Opus 5.5 (max, 111 tokens/s) and ~29 minutes for Gemini 4 Carbon (high, 71 tokens/s), because it uses ~50k output tokens per task against ~120k to ~170k.
➤ API default effort cuts Time per Task by ~3 to 5x and Cost per Task by ~2 to 4x for GPT-6 Astra, GPT-6.1 Sol and Claude Opus 5.5: all three default to medium effort, which scores ~5 to 11 points lower on the Intelligence Index in exchange. Claude Opus 5.5 falls from ~26 minutes and $10.04 per task at max to ~6.8 minutes and $2.45 at medium effort.
➤ Endpoints that optimize for decode speed alone see the largest drop in End-to-End Output Speed: high decode speeds can come at the cost of latency, for example by queuing requests to run at lower batch sizes, or by holding output and sending it in a single burst. Both lengthen the wait before the first token, which End-to-End Output Speed now counts.
