LLM验证新框架:选择性精修提升医疗问答准确率

Rethinking LLM Verification: Evidence Structure, Uncertainty, and Selective Refinement

精选理由

这篇论文教你怎么用弃权信号做推理精修,不用建知识图谱,医疗问答准确率直接涨9.6个点,GPT-5.5和DeepSeek-R1都试过。

AI 摘要

arXiv新论文提出两阶段框架,针对医疗假设验证的选择题场景,仅在模型不确定时进行本体论接地,管理覆盖率与准确率的权衡。实验显示,弃权并非随机,而是反映真实不确定性,弃权预测的置信度较低。在GPT-5.5和DeepSeek-R1两个前沿模型上,该框架将问题级准确率从82.9%提升至92.5%,提升9.6个百分点;假设级准确率从92.0%提升至96.2%,提升4.2个百分点。在MedReason和MedQA基准上的实验表明,弃权可作为选择性推理精修的控制信号,无需显式构建知识图谱即可达到图谱级性能。

原文 · arXiv: OpenAI

Rethinking LLM Verification: Evidence Structure, Uncertainty, and Selective Refinement

Large language models (LLMs) often rely on shortcuts rather than systematic reasoning, raising safety concerns in medical applications. Allowing models to abstain when uncertain improves reliability but introduces a coverage accuracy tradeoff. We propose a two-stage framework for medical hypothesis verification in multiple-choice settings that manages this tradeoff through targeted ontology grounding, applied only when the model abstains. We show that abstention is not random but reflects genuine uncertainty, with abstained predictions associated with lower confidence. Across two frontier models (GPT-5.5, accessed via the Azure OpenAI API, and DeepSeek-R1), the proposed framework improves question-level accuracy by 9.6 percentage points (82.9% to 92.5%) and hypothesis-level accuracy by 4.2 percentage points (92.0% to 96.2%). Our experiments conducted on MedReason and MedQA show that abstention can be repurposed as a control signal for selective reasoning refinement, achieving knowledge-graph-level performance without explicit knowledge graph construction.

LLM验证新框架:选择性精修提升医疗问答准确率 · AI 热点