Papers

决策模型用于癌症病理报告临床数据抽取的准确性评估

Can decision models extract clinical data from cancer pathology reports? An evaluation of accuracy, calibration, and failure modes

Why we picked it

医疗文本抽取这块,Tev1 4B 在本地硬件上跑病理报告,速度是 Qwen3.5 9B 的四倍,准确率持平,概率还能当置信度用来决定哪些答案送 GLM-5.3-flash 复核。做医疗信息抽取或本地部署的开发者可以看看它的失败模式分析。

一项发表于 arXiv 的研究首次评估了决策模型(decision models)在临床文本上的表现。决策模型不生成文本,而是从固定选项中每题选一项并返回概率。研究用 Nimble 9B 和 Tev1 4B 两个开源权重决策模型,从 1,113 份口腔癌病理报告中抽取 25 个字段,并与本地运行的 Qwen3.5 9B 和云 API 的 GLM-5.3-flash 两个生成式模型对比。

Body

结果显示,Tev1 平均字段准确率为 0.917(95% CI 0.913–0.920),与 Qwen3.5 9B(0.918)持平,但速度快四倍(每份报告 15 秒对 61 秒);GLM-5.3-flash 达到 0.986。决策模型在简单二分类字段上与 GLM-5.3-flash 相当,但在测量值分箱、"未报告"类字段和复杂判断(如切缘受累)上表现较弱。Nimble 的性能劣势主要源于单次请求包含 25 个问题,改为每请求 2 个问题后准确率有所提升。

汇总来看,两个决策模型返回的概率校准良好(期望校准误差分别为 0.022 和 0.026),但无法标记单字段内的系统性错误。模拟实验显示,把概率低于 0.94 的答案升级发送给 GLM-5.3-flash,可将 Tev1 准确率提升至 0.982,其中 38% 的答案被升级处理。作者建议决策模型应在实际部署配置下验证、限定在可靠字段上使用,数值类字段需结合其他方法。

Original · arXiv cs.AI

Can decision models extract clinical data from cancer pathology reports? An evaluation of accuracy, calibration, and failure modes

Decision models are a new class of language model that, instead of generating text, choose one option per question from a fixed list and return a probability for every option. They have not yet been evaluated on clinical text. We evaluated two open-weights decision models, Nimble 9B and Tev1 4B, for extracting 25 fields from 1,113 oral cavity cancer pathology reports from a single centre, against a prospectively maintained clinical database. We compared them with two generative models: Qwen3.5 9B, run locally, and GLM-5.3-flash, run through a cloud API. Tev1 reached a mean field accuracy of 0.917 (95% CI 0.913 to 0.920), the same as Qwen3.5 9B (0.918; p = 0.26), and was four times faster (15 s against 61 s per report). Nimble reached 0.862 and GLM-5.3-flash 0.986. Both decision models matched GLM-5.3-flash on simple dichotomous categories. They were weak on measurements binned into categories, on fields whose correct answer was not reported, and on some complex decisions (e.g. involved margins). We found that the main cause of the inferior performance of Nimble's came from asking 25 questions in one request: with two questions per request, its accuracy improved. Pooled over fields, the returned probabilities were well calibrated (expected calibration error 0.022 and 0.026), but they did not flag systematic errors within single fields. A simulated strategy of sending answers with a probability below 0.94 to GLM-5.3-flash raised Tev1's accuracy to 0.982, with 38% of answers escalated. Decision models can extract explicitly stated findings from pathology reports quickly on local hardware and return useful probabilities. They should be validated in the configuration in which they will be used, restricted to the fields in which they are reliable, and combined with other methods for numeric values.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include decision models, Tev1, Qwen3.5, GLM-5.3-flash.

Source attribution: curated from arXiv cs.AI (@Indranil Mallick, Paromita Roy).