Decision 与 System One 模型在风湿病学考试题和诊断病例上的评测
Decision and System One models in rheumatology: accuracy and calibration on specialty-training examination questions and diagnostic case vignettes
拿考试题和真实诊断病例测了一批医疗 AI 模型，结果反差挺大：考试能上 93%，真病例却掉到 50%，高置信答案也没那么靠谱。
研究用两套基准评测 System One 系列模型：扩展版 RheumaMIR（172 道西班牙专科考试题）和 RheumBench（70 个英文诊断病例，含 5-18 个候选诊断）。在考试题上 Jev 准确率达 93.0%（西语），GPT-6 Sol 最高为 99.4%，GPT-6 Luna 为 97.7%。但在诊断病例上准确率整体下降，GPT-6 Sol 为 76.9%，Jev 仅 50.0%，Laya 接近随机水平（随机为 9.6%）。Jev 在病例题上出现过度自信，概率 ≥0.90 的答案正确率只有 84.6%，而 GPT-6 Luna 同样条件下达 95.5%。
Decision and System One models in rheumatology: accuracy and calibration on specialty-training examination questions and diagnostic case vignettes
Objectives: To evaluate the accuracy and calibration of Decision and System One models on rheumatology examination questions and diagnostic vignettes. Methods: Two benchmarks were analysed: 172 rheumatology questions from Spanish specialty-training examinations (expanded RheumaMIR dataset) in Spanish and English, and 70 English diagnostic vignettes from RheumBench with 5-18 candidate diagnoses, 52 of which included the reference diagnosis. Four System One models (Jev 1.13, Clef, Microsoft-Decision-1, Laya) and GPT-6 Luna (OpenAI Decisions API) were compared with GPT-6 Sol and two local generative models. The primary outcome was accuracy; secondary outcomes included calibration, discrimination, accuracy of answers with probability >=0.90 and language differences. Results: In RheumaMIR, Jev's accuracy was 93.0% in Spanish and 91.9% in English, below GPT-6 Sol (99.4%; Holm-adjusted P<=0.005); GPT-6 Luna reached 97.7%, Clef 91.9-93.0% and Microsoft-Decision-1 86.0-87.8%, without detectable language differences. Jev's answers with probability >=0.90 (76-77% of answers) were 98.5% correct. In RheumBench, accuracy was lower: GPT-6 Sol 76.9%, GPT-6 Luna 63.5%, Clef 59.6%, Jev 50.0% and Microsoft-Decision-1 40.4% (each System One model below GPT-6 Sol, Holm-adjusted P<=0.045; chance 9.6%). Jev became overconfident (mean probability 0.659; over/underconfidence index +0.159), and its answers with probability >=0.90 (25% of cases) were 84.6% correct, against 95.5% for GPT-6 Luna (42%). Laya performed near chance in both benchmarks. Conclusion: The best-performing System One models were highly accurate on examination questions but less accurate on diagnostic vignettes, remaining below GPT-6 Sol. Jev's high-confidence predictions were less reliable on vignettes, highlighting the need for task-specific evaluation and calibration.
- OpenBMB10-28 13:01原文
