InclusionMed:面向包容性医疗的 AI 评测基准体系发布
InclusionMed: Benchmarking AI for Inclusive Healthcare
一篇医疗 AI 评测基准论文,从 388 个候选里筛出 5 个靠谱基准,测下来顶级模型总分都没过 50 分。做医疗 AI 选型的团队可以直接拿来当选型参考,做评测的研究者也能从它的审计和复现流程里学到东西。
研究团队在 arXiv 发布 InclusionMed,一个面向包容性医疗的开放 AI 评测计划。其 Atlas 框架按 3 种医疗角色和 3 种 AI 能力划分 9 个格子,首批从 388 个候选基准中经结构化审计与独立复现选出 4 个公开基准,加上私有诊断推理基准 InclusionMed-DxFiles,共覆盖 5 个基准、9 格中的 3 格。
团队在 Atlas 上评测了多个前沿大语言模型,结果汇总为 InclusionMed Index,总分范围在 17.5 到 49.1(满分 100)之间。Index 还按医疗角色和 AI 能力分别打分,发现模型排名会随角色或能力维度变化。任务级标签分析显示,Index 排名前 10 的模型在患者沟通上的得分均高于临床决策。总分相近的模型,其估算推理成本差异也很大。
论文指出当前基准覆盖仍不均衡,呼吁以更多元的医疗需求来定义和衡量医疗 AI 进展,并通过社区开放协作扩展评测覆盖。
InclusionMed: Benchmarking AI for Inclusive Healthcare
Advancing medical AI for inclusive healthcare requires evaluation across diverse healthcare roles, tasks, and care settings, but benchmark coverage remains uneven. We introduce InclusionMed, an open medical AI evaluation initiative for inclusive healthcare. The InclusionMed Atlas organizes benchmarks across three healthcare roles and three AI capabilities. For the first release, we selected four public benchmarks from 388 candidates through structured audits and independent reproduction. With InclusionMed-DxFiles, our private diagnostic reasoning benchmark, the Atlas contains five benchmarks covering three of its nine role--capability cells. We evaluated frontier large language models on the Atlas and aggregated their results into the InclusionMed Index, with overall scores ranging from 17.5 to 49.1 out of 100. The Index also reports scores by healthcare role and AI capability, showing that a model's rank varied with the role or capability considered. A within-benchmark analysis using task-level tags showed that the ten highest-ranked models in the Index all scored higher on patient communication than on clinical decision-making. Models with similar Index scores also differed substantially in estimated inference cost. These findings highlight the need for evaluation covering more diverse healthcare needs and for context-specific model selection. Through open collaboration and community contributions, InclusionMed seeks to ensure that diverse healthcare needs worldwide shape how medical AI progress is defined and measured.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include InclusionMed, medical AI, benchmarks, clinical decision making.
Source attribution: curated from arXiv cs.AI (@ The InclusionMed Collaboration).