Jev Router Experiment Shows Benchmark Limits in Agent Harness Testing
Independent harness tests like this show why vendor benchmark numbers often fail to predict task-level performance for your own workflows. Builders evaluating model routers or agent harnesses can take the concrete lesson that results depend heavily on the test setup, so replicating with your own workloads matters more than leaderboard rankings. The System One versus System Two framing offers a practical lens for deciding when to route cheaply and when to pay for heavier reasoning.
Elvis Saravia (omarsar0) ran a custom harness experiment measuring agent routing performance, finding Jev Router comparable to GPT-6 Astra low in his tests, though with longer runtimes.
He notes the results come from his own harness and may not generalize to other tasks.He contrasts his setup with Theo's scoped tests using DeepSWE, arguing that public benchmark numbers say little about real-world performance.
Saravia describes Jev as a System One model that should be combined with, rather than compared to, System Two models like GPT-6 Astra.
He calls routing largely unsolved and hopes OpenRouter will publish more on its measurement methods.
An interesting discussion is happening in the comments. I think it's worth highlighting. I ran this little experiment because I wanted to measure something I use agents for in the real world. The numbers are promising, but I must acknowledge they may not generalize as well to every task. That needs to be said. It works for my custom harness, but it may not generalize to all cases. A lesson here is that public benchmark numbers don't mean much. What may work for you may not work for me, and vice versa. Theo brought up his scoped tests that he did with DeepSWE. You can see the Jev Router performance is comparable to GPT-6 Astra low, but at the expense of a longer run. Mine and his are very different tests measuring different things. And with some disagreements and agreements. But there are so many ways to test and compare this. I also think the reasoning effort is an interesting use case for routing. Jev, as I have been posting, is a System One model and shouldn't be compared with System Two models. Jev doesn't have the capabilities of, say, a GPT-6 Astra, and so will be limited in what you can do with it. I think the more interesting experiment is combining them, hence my post. Even more interesting is potentially a hybrid model, as I shared in a previous post. Routing is largely unsolved, but we all want it. So I don't think we can dismiss ideas based on our little benchmark tests that say nothing about the real world. I am learning that it's getting extremely hard to measure and compare things these days. It's expensive and time-consuming. I really appreciate people like Theo for testing things and sharing his findings, which is what I tried to do too. I hope OpenRouter can share more on how they are measuring this. We also need to keep an open mind and discuss these ideas openly. We shouldn't encourage dismissing ideas so quickly, especially not in the world of harness engineering. I remain positive and believe these discussions will yield better solutions to these problems. I hope to run even more tests with my custom harness and potentially measure more of these things. As an independent researcher, I will continue sharing these ideas openly and hope to spark interesting discussions around harness engineering. Thanks to those who took the time to reply.📊 1 ⚡
This report is categorized under AITOP's "How-to" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include Jev Router, GPT-6 Astra, OpenRouter, agents.
Source attribution: curated from elvis (@omarsar0). Cross-source corroboration: 11 independent sources have covered this development (including Artificial Analysis, @OpenAIDevs, Y Combinator), confirming industry-wide signal strength.
