Simon Willison做了smevals评估工具，YAML写测试，一条命令可跑GPT-5.5和Claude Opus 4.6对比，生成报告。想搭评估套件可试试。
Simon Willison与Jesse Vincent的Prime Radiant实验室合作开发了smevals评估套件。该工具通过YAML文件定义评估，支持用uvx smevals run -m gpt-5.5 -m claude-opus-4.6同时运行多个模型对比。运行和评分分离，可用uvx smevals grade对结果按预设检查项打分。最后通过smevals serve或smevals build生成可分享的HTML报告。作者称这是其第三代评估方案，将用于自己的项目。
smevals - a small eval suite for evaluating models, prompts, and harnesses
smevals - a small eval suite for evaluating models, prompts, and harnesses I've been working with Jesse Vincent's Prime Radiant applied AI research lab building out this evals framework to help answer questions about the capabilities of different models. The result is smevals , a new tool for running small eval suites across different model configurations and grading the results. This blog entry describes the tool in detail. Here's the 10 second version: Tell your coding agent to run uvx smevals docs to learn the tool (this outputs the README ) Then tell it to build you an eval suite Once you've created an eval - which takes the form of a directory with some YAML files - you can run it against models like this: uvx smevals run path-to-eval/ -m gpt-5.5 -m claude-opus-4.6 Runs are treated separately from grading operations - you can grade your runs (against your defined set of checks) using: uvx smevals grade path-to-eval/ Then you can run a localhost web server to explore the results: uvx smevals serve path-to-eval/ Or run the smevals build command to build that report as static HTML, which you can then host anywhere. Here's an example showing an eval suite I built to evaluate how well models can write haikus. I've been trying to figure out an approach I like for evals for several years now. smevals is my third iteration on the idea and it feels right to me. I'm looking forward to expanding this more in the future, as well as pointing it at some of my own projects. Tags: projects , ai , generative-ai , llms , llm , evals , jesse-vincent