苹果出了个新工具，能把MCP规范自动变成测试套件，比人工建基准快多了
苹果发布Agent Seer论文，可将MCP规范转化为评估套件。该工具从单一MCP规范出发，无需示例、实时工具访问或领域特定调优即可合成多轮智能体测试场景。研究团队在七个不同领域的MCP规范上进行了测试，在小规模和中规模规范上实现了完整的工具覆盖。参数架构复杂度比工具套件大小更能预测质量差异。
Banger paper from Apple. If you build MCP servers, this can help you turn your specification into a...
Banger paper from Apple. If you build MCP servers, this can help you turn your specification into an evaluation suite. (bookmark it) It's actually a really neat idea that's easy to implement. And it showcases the awesomeness of MCP. Agent Seer starts from a single MCP spec and synthesizes multi-turn agent test scenarios with no examples, no live tool access, and no domain-specific tuning. Function names, natural-language descriptions and typed parameter schemas already carry enough semantics to generate graded scenarios with synthetic tool outputs, which then expand into mock-data-grounded dialogues. Hand-built agent benchmarks demand deep domain expertise, do not scale across tool ecosystems, and go stale as soon as an API changes. Generating them from the live spec keeps pace with the ecosystem instead. They ran it on seven MCP specifications spanning different domains and suite sizes, with complete tool coverage on small and medium specs. Parameter schema complexity predicts quality variation far better than tool-suite size does. And argument value accuracy is the dominant failure mode, a sub-dimension that coarse name-match tool-calling metrics cannot see at all. Paper: arxiv.org/abs/2608.26133 Chat with Paper: academy.dair.ai/papers/agent-s… 💬 10 🔄 1 ❤️ 25 👀 2318 📊 15 ⚡