How-toSelected9月21日 00:39

Jev as a judge

"jev as a judge" cheap and fast semantic verifiers! great for evals - especially online evals where...

Harrison Chase@hwchase17

Post

Why we picked it

For developers working with LLMs, Jev offers a potentially cheaper and faster alternative to traditional LLM-based evaluation methods.

A new tool called Jev is introduced as a cheap and fast semantic verifier for evaluating LLMs. It was tested against LLM judges on accuracy, repeatability, latency, and cost. The results suggest it could be a new approach for agent evaluation.

Original · Harrison Chase

"jev as a judge" cheap and fast semantic verifiers! great for evals - especially online evals where...

"jev as a judge" cheap and fast semantic verifiers! great for evals - especially online evals where you want to grade LOTS of traces LangChain @LangChain We tested Jev against LLM judges on accuracy, repeatability, latency, and cost to see whether System One models could offer a new approach to agent evaluation. x.com/i/article/2101… 🔗 View Quoted Tweet 💬 4 🔄 0 ❤️ 15 👀 877 📊 5 ⚡

#Jev #LLM evaluation #agent evaluation #prompt engineering

Keep reading

Recent stories with the same tag or section

Models Builders09:47

Chinese content扩散语言模型结合 Jev 的新思路引发关注

有人把扩散语言模型和 Jev 拼到一起做生成，思路挺新鲜，想追细节的可以去原推文留言问问。

#扩散模型 #LLM #Jev

Xianbao Qian@Xianbao_QIANSource

Papers BuildersMulti-sourceSelected14:19

DecisionRAG: Cornell paper replaces retrievers with a decision model over document trees

The interesting shift is that retrieval becomes a tree-traversal decision problem instead of an embedding-similarity problem, so latency scales with tree depth rather than corpus size, with billion-paragraph corpora theoretically needing around five decisions. Teams paying for a reranker stage, or struggling with weak first-stage recall on help centres and document corpora, can test the tree-plus-decision-model setup; on NQ a dense retrieval stage can optionally be mixed in to recover over-pruned evidence. It competes with Qwen3-Reranker-8B pipelines at lower cost.

#DecisionRAG #Jev #RAG #Qwen3

Event

shao__meng@shao__meng3 sourcesSource

Papers10:46

Chinese contentSatOps-Decision：90.3 毫秒低延迟星载卫星通信决策模型

一篇把判别式决策模型搬到卫星通信上的论文，单次前向 90 毫秒出决策，准确率还比生成式基线高出一大截，做边缘部署的可以看看。

#SatOps-Decision #Jev #Laya #推理模型

arXiv cs.LG@Lifan Mei et al. (4)Source

Papers BuildersMulti-sourceSelected18:26

Jev Matches Frontier LLM Accuracy in RAG Claim Verification

Headline accuracy is the wrong metric when choosing a RAG claim checker, because a verifier can look accurate while waving through a large share of unsupported claims. Teams building retrieval-backed products should benchmark false-approval rates alongside cost before wiring in a cheap verifier, and cascade designs may not save more than random routing. The finding applies to any RAG stack where a wrong approval is costlier than a wrong rejection.

#RAG #verifier models #Jev #GPT-6 Astra

Event