研究追踪 LLM 生成 UML 类图时幻觉的来源
Tracing Hallucinations in LLM Reasoning for UML Class Diagram Generation: An Empirical Taxonomy and Mitigation Study
如果你用 LLM 画 UML 类图老出错，这篇把幻觉发生在哪一步讲清楚了，还给了实测能涨 4 个点 F1 的提示词写法。
一项研究对 LLM 生成 UML 类图过程中的幻觉做了分类，把幻觉拆为 Prompt Faithfulness、Code Faithfulness、Reasoning Quality 三个维度、11 个子类。分类体系来自 60 条人工标注的 DeepSeek-R1 推理轨迹，并在 120 条 DeepSeek-R1 和 20 条 DeepSeek-V4-Pro 轨迹上验证。DeepSeek-R1 中 Code Faithfulness 幻觉最多，DeepSeek-V4-Pro 则以 Prompt Faithfulness 为主，失败模式因模型而异。在 13 个 Golden UML ModelSet 用例上，对 Qwen3 系列模型在提示词中加入 Prompt Faithfulness 和 Reasoning Quality 约束后，平均 F1 提升 4.0 个百分点。
Tracing Hallucinations in LLM Reasoning for UML Class Diagram Generation: An Empirical Taxonomy and Mitigation Study
Large Language Models (LLMs) can generate UML class diagrams from natural language requirements, but hallucinations may compromise the correctness and completeness of the resulting diagrams. Existing evaluations mostly assess the final diagram, without examining the intermediate traces of reasoning-capable LLMs, which can reveal where these hallucinations arise. We develop an empirical taxonomy of eleven hallucination sub-types organised into three dimensions: Prompt Faithfulness (deviations between requirements and reasoning), Code Faithfulness (discrepancies between reasoning and generated PlantUML code), and Reasoning Quality (problems within the reasoning). The taxonomy was derived from 60 manually annotated DeepSeek-R1 traces, applied to 120 further DeepSeek-R1 traces through a dual-LLM annotation pipeline, and examined on 20 DeepSeek-V4-Pro traces. Code Faithfulness was the most frequent dimension for DeepSeek-R1 in both datasets, whereas Prompt Faithfulness dominated for DeepSeek-V4-Pro, suggesting model-dependent failure patterns. In the manually annotated traces, reasoning-trace length showed no significant monotonic association with hallucination frequency, whereas requirement complexity was negatively associated with it, particularly for Prompt Faithfulness errors. In a mitigation experiment on 13 Golden UML ModelSet cases with three Qwen3-based models, adding Prompt Faithfulness and Reasoning Quality constraints to the generation prompt raised the pooled mean F1 score by 4.0 percentage points, whereas a post-hoc Code Faithfulness review provided no consistent further benefit. Relationship modelling remained the lowest-scoring element throughout. These results suggest that localising hallucinations within the generation process can inform targeted, prompt-based mitigation. Whether these findings generalise beyond the DeepSeek and Qwen3 model families remains to be tested.
