想测AI罕见病诊断能力？GraphRareBench用2365个案例和18093对混淆，专门检测模型能不能分清真病和假病。
GraphRareBench是一个保留来源的基准，包含2,365个本体派生案例和18,093对目标-混淆对。在237个案例的基因组分测试集上，监督排序器MRR范围为0.640-0.740，目标-混淆准确率为0.898-0.916。基于Agents-A1和DeepSeek-V4-Flash的智能体MRR分别为0.746和0.718，但证据覆盖率差异达0.561。22.1%到43.7%的成功案例中仍存在硬混淆者排名高于目标。该基准可用于评估模型在完全集检索和硬混淆区分上的表现。
GraphRareBench: An Auditable Graph-Evidence Benchmark for Phenotype-Driven Rare-Disease Diagnosis
Phenotype-driven diagnostic benchmarks usually report the rank of the reference disease, but they rarely reveal which plausible alternatives are ranked above it or what evidence a tool-using model examines before making its decision. We introduce GraphRareBench, a provenance-preserving benchmark containing 2,365 ontology-derived cases and 18,093 target-confounder pairs. Each case includes a coarsened HPO query, a fixed candidate pool, graph-defined hard confounders, and source-linked evidence records. On the 237-case gene-component-disjoint test split, supervised rankers using a shared 21-feature interface achieved MRRs ranging from 0.640 to 0.740 and case-averaged target-over-confounder accuracies ranging from 0.898 to 0.916. Agents instantiated with Agents-A1 and DeepSeek-V4-Flash achieved MRRs of 0.746 and 0.718, respectively. Their paired MRR difference was not statistically significant, whereas their target-evidence coverage differed by 0.561. Together with the observation that 22.1% to 43.7% of selected Hit@10 successes still ranked at least one graph-defined hard confounder above the target, these results indicate that full-pool retrieval, hard-confounder discrimination, and observable evidence access capture complementary aspects of model behavior. GraphRareBench therefore provides a foundation for more transparent and evidence-aware evaluation of phenotype-driven diagnostic systems. Code and data are available at https://github.com/GUI0609/GraphRareBench.