医疗微调不降风险只重塑风险：六模型评测框架发布
Medical Fine-Tuning Reshapes, Not Reduces, Patient-Facing AI Risk: A Rubric-Based, LLM-Judge Evaluation Framework
医患问答光看幻觉率不够，这篇用六类风险语言给六个 7-9B 模型打分，发现医疗微调只是换了个风险方向，评测代码也开源了。
arXiv 论文提出基于六类风险语言的评测框架，涵盖治疗指令、禁忌、剂量、紧急分诊、高警示药物和过度自信断言，由三个 LLM 评委组成集成打分。团队用 K-QA 基准中 200 条真实患者问题评测六个 7-9B 模型，包括 Mistral-7B、Phi-3.5-mini、Llama-3.1-8B 与 BioMistral-7B、OpenBioLLM-Llama3-8B、Med42-Llama3-8B。结果显示风险语言与事实幻觉率统计独立（Spearman ρ=-0.065，独立性检验 p=0.24）。医疗微调不降低总风险而是重塑：BioMistral 和 OpenBioLLM 压制高警示药物提及但放大过度自信断言，Med42 只压制剂量表述。四个语言锚定类别的 Krippendorff α 为 0.53-0.71，两个依赖临床判断的类别仅 0.19 和 0.31，rubric、评委提示词和评测代码已开源。
Medical Fine-Tuning Reshapes, Not Reduces, Patient-Facing AI Risk: A Rubric-Based, LLM-Judge Evaluation Framework
Patient-facing medical question answering with large language models (LLMs) is moving from research demonstrations to production deployments, yet evaluation remains anchored on factual correctness. Factuality is necessary but not sufficient: a response can be factually grounded and still issue a directive that, if acted on, would harm the patient. We introduce a risk-sensitive evaluation framework that scores responses by the presence and severity of six categories of risk-bearing language (treatment directives, contraindications, dosage expressions, urgency-triage cues, high-alert medication mentions, and overconfident assertions), anchored on published clinical harm taxonomies. Severity is classified by a three-judge LLM ensemble. We apply this framework to six 7-9B instruction-tuned models, three general (Mistral-7B, Phi-3.5-mini, Llama-3.1-8B) and three medically adapted (BioMistral-7B, OpenBioLLM-Llama3-8B, Med42-Llama3-8B), on 200 real patient questions from the K-QA benchmark. Two findings emerge. First, ensemble risk-bearing language is statistically independent of factual hallucination rate (pooled Spearman $ρ= -0.065$, $n = 300$, 95% CI $[-0.18, +0.05]$; distance correlation $= 0.10$, independence test $p = 0.24$). Second, medical fine-tuning does not reduce aggregate risk but reshapes it: continued-pretraining adaptations (BioMistral, OpenBioLLM) consistently suppress high-alert medication mentions while amplifying overconfident assertions, whereas clinical instruction-tuning (Med42) suppresses dosing only, without amplifying overconfidence. Per-category Krippendorff $α$ is fair to good for the four linguistically anchored categories (0.53-0.71) and low for the two clinical-judgment-dependent categories (0.19, 0.31), indicating where automatic safety evaluation is and is not yet reliable. We release the rubric, judge prompts, and evaluation code.
