Jev tested as a fuzzy linter inside agent harnesses
The practical trick is the two-tier confidence output: routing medium-confidence flags back to the agent for verification doubles catch rate without materially inflating false positives. Developers building agent harnesses can offload cheap, deterministic checks like linting to small System One models instead of paying frontier-model prices for every call. DAIR.AI has published related resources at academy.dair.ai for those wanting to replicate the setup.
Michael Thiessen has been experimenting with Jev as a fuzzy linter that runs after edits in an agent harness.
He converted coding guidelines into tiny single-call rules that do not require extra context or reasoning, then filtered out rules too complex for the model.
He built a synthetic eval with tool calls to exercise each rule, plus a held-out set to avoid overfitting.Jev returns confidence scores, and adding a medium-confidence tier that prompts the agent to double-check roughly doubled detection coverage with only a small rise in false positives.
Elvis Saravia of DAIR.AI highlighted the setup as a good example of pairing System One models with System Two models in a harness.
This is brilliant. Recommended reading. It's a great example of how you could potentially build more effective harnesses using System One and System Two models. It explores Jev's potential as a fuzzy linter. Before you dismiss it, check out the setup. I like this post because it provides a few details of a good setup for testing these ideas. It scopes the problem well, like filtering out rules that are easy to judge (i.e., don't require extra context/resoning). That's important because System One models like Jev aren't built for reasoning-heavy tasks. But it doesn't mean that they aren't more deterministic operations where Jev could help scale efficiency and performance. You just have to look closer at the harness. In the harness, a System Two model typically controls all the components and decisions that you could potentially offload to a System One model. It's aligned with what ideas I shared here: academy.dair.ai/resources/jev-… It's becoming extremely obvious that you don't need frontier models for everything. You are likely paying a premium for something you don't need. I really hope we get more evals and benchmarks to measure these things because there is a thread worth pulling here. Michael Thiessen @MichaelThiessen I’ve been experimenting a bit with Jev as a fuzzy linter that runs after edits in your agent harness: looks very promising so far in my evals. First I converted all of our coding guidelines to tiny rules that are very easy to judge. Then, only kept the ones that don’t require extra context/files/reasoning. I want it to be a single call, super fast, and Jev isn’t smart enough to handle complex or ambiguous rules. I then created a synthetic eval that has different kinds of tool calls to exercise and test every one of those rules. Also a held out set that we don’t train on so we don’t accidentally overfit (ie benchmax). One cool thing that seems to work well: Jev returns confidence, so in addition to high confidence violations, we can also return medium confidence and get the agent to double check. Adding that lower confidence tier seems to double what it catches, and only increases false positives a little. I still have yet to test this inside an actual agent to see how the agent reacts to this feedback. Hopefully next week I can get a decent eval going and see how well this works E2E. 🔗 View Quoted Tweet📊 5 ⚡
This report is categorized under AITOP's "How-to" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Jev, DAIR.AI, agents, coding assistants.
Source attribution: curated from elvis (@omarsar0).
