Latest

The benchmark suggests a practical routing rule: swap general-purpose LLM calls for Jev only on narrow, high-frequency decisions like stopping checks, and keep dedicated rerankers like Voyage rerank-3 for re-ranking. Teams running RAG pipelines can use the latency and cost figures to decide where Jev cuts spend without hurting recall. Full results are on Milvus's blog.

Milvus@milvusioSource

The takeaway is that small models can beat larger ones when given enough parallel samples and a verifier, which reframes inference-time compute as the practical lever for builders without trillion-parameter budgets. Engineers designing agent pipelines should note the coverage-versus-pass@k distinction when choosing between repeated sampling and reasoning models like o1. Stanford publishing the full lecture makes a frontier-lab curriculum freely accessible to self-taught practitioners.

shao__meng@shao__mengSource

The finding inverts common prompt-engineering instinct: adding reasoning requests degrades agent decisions while plain factual statements about rules improve them. Developers building bidding, negotiation or matching agents can copy established market-design heuristics instead of inventing new prompt tricks. Judging agents by their written plans is also misleading, because the models' choices improved even when their stated reasoning did not reflect the gains.

rohanpaul_ai@rohanpaul_aiSource

Running a 27B model locally makes arithmetic-accuracy testing repeatable and free of API costs, and the reasoning-versus-non-reasoning comparison shows whether extended thinking actually helps on mechanical tasks. Developers choosing models for local deployment can use this as a practical benchmark. It also complements Fraser's earlier GPT-4o results as a direct point of comparison.

官方账号Simon Willison@simonwSource

A 27B open-weights model running locally now solves multi-digit addition-in-words at near-perfect accuracy, a task GPT-4o fumbled two years ago. Practitioners evaluating small local models for arithmetic-adjacent workflows get a concrete benchmark on consumer-tier DGX Spark hardware. Comparisons across reasoning-on and reasoning-off settings make it easy to judge whether enabling reasoning is worth the latency.

The cost gap, 62x cheaper than LLM judges, makes calibrated decision models a practical replacement for expensive evaluation pipelines. It also surfaces label errors in three existing benchmarks, which anyone relying on published alignment scores should note. Thresholds do not transfer between benchmarks, so teams adopting this should fit one on roughly 10 labelled items, which lifts median F1 from 0.706 to 0.793.

elvis@omarsar03 sourcesSource

The benchmarks show where cheap small models can replace larger ones in RAG pipelines: stopping decisions and relationship filtering see large latency and cost savings with matched recall. Engineers building retrieval systems can use the published code to pick where to route between Jev, GPT-5-mini and Voyage rerank-3. The evaluation includes Chinese-language queries, useful for teams serving bilingual users.

Milvus@milvusioSource