想看看AI在基因组监测上多不靠谱？新基准BioSecBench-Surveillance测了16个模型，最好才一半正确，细节值得细读。
新基准BioSecBench-Surveillance包含100个评估任务，覆盖7个类别，从分类到基因工程检测。16个模型-工具组合在3962次尝试中，最佳配置Opus 4.8 with PI仅达50.2%准确率，与GPT-5.5 with Codex并列。Opus 4.7 with PI为49.6%，Sonnet 4.6 with PI为48.6%。即使调用正确工作流，错误仍源于参考序列、阈值、过滤器等选择失误。该基准为衡量下次疫情爆发时AI代理的可信度提供了标准。
BioSecBench-Surveillance: A Verifiable Benchmark for AI Agents in Pathogen Genomic Surveillance
As pathogen genomic surveillance scales, the bottleneck is shifting from data generation to analysis. We present BioSecBench-Surveillance, a verifiable benchmark of 100 evaluations testing whether AI agents can infer the right analysis pipeline from raw sequencing data and surveillance context. Each evaluation gives an agent only the data and context a human analyst would have, then grades its structured answer deterministically. The tasks span seven categories, from taxonomic classification to genetic-engineering detection, across diverse sample types and sequencing technologies. Across 3,962 gradable attempts from sixteen model-harness pairs, the strongest configuration cleared only about half. Opus 4.8 with PI led at 50.2 percent, with a 95 percent confidence interval of 40.1 to 60.3 percent across 83 evaluations, tied with GPT-5.5 with Codex at 50.2 percent, with a 95 percent confidence interval of 40.8 to 59.6 percent, followed by Opus 4.7 with PI at 49.6 percent, with a 95 percent confidence interval of 40.0 to 59.2 percent, and Sonnet 4.6 with PI at 48.6 percent, with a 95 percent confidence interval of 38.9 to 58.3 percent. Even when agents invoked the correct workflows, their mistakes came from the choices around them, such as which references, thresholds, filters, and normalization to apply. BioSecBench-Surveillance provides a standard for measuring whether agents can be trusted to perform genomic surveillance when the next outbreak arrives.
- Aravind Srinivas03:36原文