DiagFlowBench这个新基准专门用来测语言模型在操作维护场景里，能不能识别用户问跑题的问题。10个模型测下来，差得挺大，而且那种看似合理但不对的答案最危险。论文值得一看。
新基准DiagFlowBench包含50张工业诊断流程图，转化为1676轮多轮对话，对比合规与偏离流程的输入。评估10个商业和开源模型发现，模型在识别超范围输入时表现差异大，常见错误是选择真实但上下文不合适的步骤。该基准揭示了基于文档的对话系统在输入偏离时容易被看似合理但错误的建议误导的脆弱性。
DiagFlowBench: Evaluating How Language Models Handle Off-Procedure Inputs in Grounded Diagnostic Dialogue
Language models increasingly serve as advisory systems in maintenance operations. To prevent hallucination, recent systems ground these models in procedural documentation to constrain them to approved steps. In practice, however, operator queries frequently stray from this path, requiring models to recognise out-of-scope inputs mid-conversation, a dynamic that current benchmarks rarely prioritise. We introduce DiagFlowBench, a dataset of 50 industrial diagnostic flowcharts from a consumer manufacturer converted into 1,676 multi-turn conversations that contrast compliant with out-of-scope utterances. Evaluating a panel of ten commercial and open-weight models reveals high variability in abstention rates, with models commonly selecting a real but contextually inadequate step rather than fabricating facts. The inherent plausibility and authority of this mapped but wrong advice exposes a challenging vulnerability for grounding systems.