做通用智能体开发的团队，评估策略往往被低估——LangChain 提出的两套测试集思路解决了成本与覆盖的平衡问题，建议直接参考这个框架来优化自己的评测流程。
LangChain 指出通用智能体的评估不能仅靠单一测试集，需要区分基准评估和测试覆盖评估。基准评估包含约100个测试用例，覆盖常见场景，用于快速判断智能体是否正常工作。测试覆盖评估则包含多达500个以上用例，覆盖每个任务的多种用户提示和路径，确保全面性。由于全面评估成本高昂（每次可能数千美元），无法频繁运行，因此需要小规模基准集用于持续集成。该观点强调了通用智能体评估的复杂性，并暗示未来可能发布更详细的博客。
Evals are not as simple as just a single set of tests, especially for general purpose agents. See on...
Evals are not as simple as just a single set of tests, especially for general purpose agents. See one idea for how we're thinking about grouping them: Brace @BraceSproul I've been thinking a lot about the two different groups of evals you need in general agents/agents which handle broad tasks: 1. Benchmark evals - this is a suite of up to 100 eval cases which test the happy paths of your agent, and its most common use cases. This isn't that comprehensive, but covers enough to where you can use it to quickly judge how well your agent handles tasks 2. Test coverage evals - this is a much more detailed suite (maybe up to 500, or more individual cases) that covers every single task you want your agent to be able to handle. It doesn't just include single tests for tasks, but multiple tests per use case, all with slightly different user prompting/tragectories There needs to be two suites for a few reasons: - general agents have so many use cases, to accurately test them, and have confidence it preforms well on everything you want to support, you need many evals for each workflow - the comprehensive eval suite will become too expensive to run on any sort of recurring basis (let alone ci) think $1000's per run, esp if you're supporting multiple models. so you need a smaller suite (the benchmark eval) to quickly gauge whether or not your agent works on code changes - in general agents, agents can preform the same tasks, but via very different paths. the final result is all the user cares about, but the intermediate steps can look very different. if your eval suite doesn't cover multiple paths to reach the same result, you can't be confident your agent will actually work well in all real world scenarios your users put your agent into there's a lot more nuance here, so maybe i'll write a longer blog post on it, and how we're thinking about maintaining/building eval suites this large... 🔗 View Quoted Tweet 💬 0 🔄 1 ❤️ 6 👀 1385 📊 1 ⚡