做长时 AI Agent 评估的开发者终于有了实用技巧——拆解任务后评估效率大幅提升，建议直接参考这个流程优化你的评估策略。
LangChain 的 Palash Shah 分享了一种针对长时运行 AI Agent 的评估方法。核心思路是将复杂的评估任务拆解成更小、更易处理的子任务，这样不仅便于人类理解，也更容易让 LLM 自身进行评估。他举例说明，对于运行超过 30 分钟的 Agent，通过从追踪中提取推理过程，找出特定行为的根本原因，然后重建简化版的评估场景。这种方法可以快速测试提示词调整的效果，而无需每次都运行完整的长时间评估。
Breaking down the eval task into easy to process chunks not only makes it easy for us humans to unde...
Breaking down the eval task into easy to process chunks not only makes it easy for us humans to understand, but also easier for LLMs to eval: Palash Shah @palashshah you can condense long horizon evals with agents into smaller subsets that still let you test intended behavior. i'm currently evaluating an agent that runs for 30+ minutes, and analyzes thousands of traces at a time. here's my process: if you're evaluating whether X impacts Y agent output, oftentimes a lot of the information that exists in X isn't relevant to the decision of Y. i extract the reasoning out of trace, and then figure out what is the cause of a specific behavior. then, i know what situation i need to re-create when setting up my eval. and as a result, i can create a much smaller/simpler version of the long horizon eval that i can quickly use to figure out what i need to change in my prompts to get the behavior i want. 🔗 View Quoted Tweet 💬 3 🔄 1 ❤️ 7 👀 961 📊 3 ⚡