Papers

临床 AI 审计研究:诊断正确不等于路径安全,修正反馈还会引入新风险

Right Diagnosis, Unsafe Route: Auditing Clinical AI Decision Trajectories and Corrective Oversight for Validity and Potential Harm

Why we picked it

这篇研究做了一件很少人做的事:不只看 AI 诊断对不对,还逐节点审计治疗路径会不会伤到病人。300 个病例里过半"诊断对了但路上有危险决策",做医疗 AI 落地或医院信息化的人应该把这套审计方法记下来。

一项发表在 arXiv 的研究提出"轨迹审计"方法,对临床 AI 的逐步决策过程做安全性评估,而不只看最终诊断是否正确。研究在一家医院心脏病队列、MIMIC-IV 和公开 ASCENT 基准共 300 个病例上展开,每个数据源最多测试 7 个 AI 系统。结果显示:在医院数据上,55.2% 最终诊断可接受的轨迹中途出现过中等或严重潜在伤害决策,24.8% 出现过临床不可接受决策;MIMIC-IV 和 ASCENT 上的伤害率分别为 76.2% 和 39.6%。

Body

方法上,轨迹审计逐检查点回放真实入院病例,让系统只用当时已释放的证据做决策,并由 LLM 评审对每项检查、治疗和处置分别打"临床有效性"和"潜在伤害"两个分数,再用医生三重标注加资深裁定的参考标准校准。监督审计则在首个可行动错误处分叉轨迹,衡量纠正反馈能否修复错误。

结果还揭示纠正本身有代价:结构化纠正修复了 73.6% 的首错,远高于单纯拒绝的 24.8%,但纠正后有 30.2% 的分支出现新的潜在不安全决策,拒绝后该比例为 19.7%。对部署临床 AI 的医院而言,这意味着上线前应审计中间决策路径,而非只验收最终诊断准确率。

Original · arXiv cs.AI

Right Diagnosis, Unsafe Route: Auditing Clinical AI Decision Trajectories and Corrective Oversight for Validity and Potential Harm

A clinical AI system can reach a correct final diagnosis through intermediate decisions that could have harmed the patient, and can repair an error while introducing new risks. Most evaluations score final diagnoses; even stepwise benchmarks, which release a case's evidence over time, score whether systems reach the right diagnosis or gather the right information, not whether the decisions a patient passes through are safe. We make patient safety the target of stepwise evaluation. A trajectory audit replays real admissions checkpoint by checkpoint, lets each system decide with only the evidence released so far, and rates every test, treatment and disposition for its potential harm to the patient and, separately, for clinical validity, against the documented final diagnosis and physician-written disease rubrics yet on the evidence available when the decision was made. An LLM judge scores every decision, and a physician reference standard (triple annotation and senior adjudication of a stratified sample) calibrates the rates. An oversight audit branches each trajectory at its first actionable error and measures, on the same branches, whether corrective feedback repairs the error and whether it introduces new harm. Across 300 cases from a hospital cardiology cohort, MIMIC-IV and the public ASCENT benchmark, with up to seven AI systems per source, 55.2% of hospital trajectories reaching an acceptable final diagnosis passed through a decision with moderate or severe potential harm and 24.8% through a clinically unacceptable one; the harm rates were 76.2% on MIMIC-IV and 39.6% on ASCENT. Structured corrections resolved 73.6% of first errors versus 24.8% after bare rejection, yet new potentially unsafe decisions followed 30.2% of corrections versus 19.7% of rejections. A correct final diagnosis does not certify a safe trajectory, and a successful correction does not certify a safe recovery.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Key entities and technical topics under continuous tracking include clinical AI, MIMIC-IV, LLM evaluation.

Source attribution: curated from arXiv cs.AI (@Guanqiao Chen, Wei Jiang, Shu Zhang, Qian Lin, Guanxu Yu, Siran Li, Chunhui Wang, Tianyu Shi).