轨迹评估需多维度考量
trajectory labeling is really several questions, not one pass/fail - nice framing here that's how J...
LangSmith的Jev框架将轨迹评估拆分为多个独立维度，解决了RL研究中一个被忽视的问题。
LangSmith评估系统中的Jev框架将轨迹评估拆分为多个独立问题。每个轨迹需评估任务难度、解决方案正确性、设计选择多样性及思维方向正确性。这种多维度评估方法解决了RL研究中一个被忽视的方面，但大规模应用仍需大幅降低token成本。
trajectory labeling is really several questions, not one pass/fail - nice framing here that's how J...
trajectory labeling is really several questions, not one pass/fail - nice framing here that's how Jev as a judge works in LangSmith evals: difficulty and correctness each get their own typed answer, in one pass, on every trace langchain.com/blog/jev-is-no… x.com/xennygrimmato_… Vaibhav Tulsyan @xennygrimmato_ This is a very clever usecase for Jev-like decision models. Trajectory labelling at scale is key for RSI. Why is that? Every trajectory needs to be assessed across several dimensions. 1. How difficult was the task? 2. Was the agent’s solution correct? 3. Did the agent consider a diverse set of design choices before deciding to pick a specific route? 4. Is the agent’s thinking directionally correct? Imagine answering these questions for millions of agents each of which spawns tens or hundreds of subagents! Token cost for trajectory labelling needs to be driven down a lot to do this at scale. This work is unique and solves a rather under-looked aspect of RL research. 🔗 View Quoted Tweet 💬 6 🔄 1 ❤️ 7 👀 1303 📊 5 ⚡
