zavoai 发布 Hospital-bench:20 个模型医院运营智能体基准实测
Introducing Hospital-bench, a new agent benchmark from @zavoai. It tests hospital operations work. ...
把 20 个模型扔进有 13 个应用、131 个工具的模拟医院里干活,GPT-6.2 Astra 拿了 62.4% 的均分,但不少模型会乱改理赔、泄露病历。做智能体评测或医疗自动化的都该看看这份成绩单。
zavoai 发布 Hospital-bench,一个评估 AI 智能体处理医院运营工作的基准。智能体在模拟健康系统内完成保险覆盖核查、预授权、理赔和病历调阅等任务,环境包含 13 个应用和 131 个工具。20 个模型共跑了 48 个任务。
结果显示 GPT-6.2 Astra 以 62.4% 的平均分领先,Claude Opus 5.5 完全通过的任务数最多(17/48)。安全性是最大挑战:模型出现过泄露受限患者病历、错误修改已支付理赔、越权操作等问题。GPT-6.2 Astra 和 Gemini 4 Neon 没有任务因造成伤害被判零分,部分模型最多有 10 个任务归零。
这组数据说明智能体在多系统复杂工作流中做到可靠且安全仍是难题,对医疗自动化落地的可行性评估有直接参考价值。
Introducing Hospital-bench, a new agent benchmark from @zavoai. It tests hospital operations work. ...
Introducing Hospital-bench, a new agent benchmark from @zavoai . It tests hospital operations work. Agents handle coverage checks, prior authorizations, claims, and records requests inside a simulated health system with 13 apps and 131 tools. 20 models ran 48 tasks. GPT-6.2 Astra led with a 62.4% mean score, while Claude Opus 5.5 fully passed the most tasks (17/48). Safety was a major challenge. Models made errors, including disclosing restricted patient records, incorrectly modifying paid claims, and acting beyond their authority. GPT-6.2 Astra and Gemini 4 Neon had no tasks zeroed due to harm, while some models had as many as 10. The results show that completing complex, multi-system workflows reliably and safely remains a major challenge for AI agents. Can Zehebi @canz101 Introducing Hospital-bench: the most comprehensive hospital operations simulation built to evaluate AI agents. Behind every hospital visit are hours of administrative work: insurance verification, prior authorizations, claims, billing, patient records, and paperwork spread across dozens of workflows. We built Hospital-bench to test whether AI agents can handle this complexity in a realistic hospital environment. Here are the results ↓ 🔗 View Quoted Tweet
This report is categorized under AITOP's "Models" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include Hospital-bench, GPT-6.2 Astra, Claude Opus 5.5, agents.
Source attribution: curated from elvis (@omarsar0).