论文精选

Structured Ignorance Certificates:让推理模型学会承认“不知道”

Calibration of Structured Ignorance Certificates for Diagnosing Unknown Unknowns in Reasoning Models

精选理由

这项研究直接戳中了LLM的“幻觉”痛点——用结构化输出让模型承认无知,做AI安全或可靠性研究的团队值得关注,尤其适合需要高可信度输出的应用场景。

AI 摘要

大语言模型常会“不懂装懂”,对超出知识边界的问题给出流畅但错误的回答。研究者提出结构化无知证书(SICs),要求模型明确输出缺失的领域交集、所需概念和检索查询,而非直接生成答案。团队构建了7347条跨领域未知-未知数据集,并用GRPO微调14B模型,使SIC输出JSON有效率达99.46%,概念特异性评分0.967。该方法证明,让模型显式表达认知边界是可学习且可衡量的能力,对提升AI可靠性和安全性有重要意义。

原文 · arXiv cs.AI

Calibration of Structured Ignorance Certificates for Diagnosing Unknown Unknowns in Reasoning Models

Large language models frequently fail in a characteristic way: rather than acknowledging ignorance, they produce fluent but incorrect answers to questions that lie beyond their knowledge boundaries. We introduce \textbf{Structured Ignorance Certificates} (SICs), a JSON-formatted output schema that demands a model explicitly name the missing domain intersection, enumerate required concepts, and propose a productive retrieval query rather than hallucinating an answer. To train models to produce high-quality SICs we construct a 7,347-sample \emph{Unknown-Unknown} (UU) dataset by prompting Qwen3-14B to stitch together questions from seven domains (physics, biology, engineering, CS, economics, medical, legal) into novel cross-domain queries that no single-domain expert could answer. We fine-tune a 14B-parameter model with Group Relative Policy Optimization (GRPO) using a composite reward that combines retrieval utility, concept specificity, and output-format validity. A paraphrase-divergence probe trained on model responses confirms that SIC-tuned outputs systematically exhibit higher unknown-unknown probability scores. Evaluation on 735 held-out UU questions achieves a 99.46\% JSON validity rate, a mean Certificate Specificity Score of 0.967, and a 3.6\% ROUGE-L improvement over the base model on retrieval-grounded generation -- demonstrating that explicit epistemic structuring is a learnable and measurable capability.

Structured Ignorance Certificates:让推理模型学会承认“不知道” · AI 热点