Papers

论文:用人格方向与知识方向探测大语言模型谎言的边界

The Geometry of Lies: Knowledge and Persona Directions for Detecting Deception in Large Language Models

Why we picked it

这篇论文做了件少见的事:把无标签人格方向测到 70B 规模,然后老实报告失败。Qwen 上失效、知识谎言基本追措辞、组合分数在留出集上输给探针——做模型安全审计或探针研究的同学值得看看这些边界在哪。

arXiv 论文(编号 2611.26545)测试了线性探针之外更廉价的无标签谎言检测方法:从诚实与欺骗角色扮演中提取 persona 方向,在 70B 规模模型上验证。作者扩展了已发布的人格库,加入“否认自己已知知识”的角色及诚实对应版本,在中间层提取方向后在 Liars' Bench 基准的四个模型上排名谎言,其中三个模型优于单数据集探针,但 Qwen 上失败,即使 Qwen 更常保持在角色中。

Body

针对“关于知识的谎言”这一难点——谎言与诚实的否认措辞可能完全相同——persona 方向主要追踪回复措辞,在 Llama 和 Qwen 上表现落入随机方向区间。作者另构建知识方向:用模型能回答的问题与无人能回答的问题(后接相同脚本化否认)对比,在两个模型上能区分谎言与相同的诚实否认,第三个模型勉强有效,第四个失败,且部分结果依赖基准措辞。两者组合的分数在全部四个模型上优于同深度的有监督探针,但在留出谎言集上落后于该探针,更丰富的角色扮演问题也没有带来帮助。

这项工作的价值在于诚实划定了线性探测方法的适用范围:跨谎言类型迁移、跨模型泛化(Qwen 失败、Gemma 仅达边缘)都存在明显限制,对依赖探针做 AI 安全审计的团队是重要的负面结果参考。

Original · arXiv cs.AI

The Geometry of Lies: Knowledge and Persona Directions for Detecting Deception in Large Language Models

Linear probes are the cheapest LLM lie detectors but often fail to transfer across kinds of lie. Label-free directions from honest and deceptive role-play are untested at 70B scale and on lies about knowledge, where a lie and an honest denial can be identical. We widen a published persona bank with roles that deny knowledge they have, plus honest counterparts. At each model's middle layer the resulting persona direction ranks lies across Liars' Bench better than single-dataset probes on three of its four models; it fails on Qwen, even when Qwen stays in role far more often. On Llama-3.3-70B-Instruct it beats directions from a random persona bank, but its gain over the published direction is within the range of random widenings. On lies about knowledge, persona directions mainly track a reply's wording: where lies and honest denials share their words, our three main contrasts stay inside the random-direction band on Llama and Qwen and reach only its edge on Gemma, though on Mistral one clears it. A knowledge direction contrasts questions the model can answer with ones nobody can, each followed by the same scripted denial. It separates lies from identical honest denials on two models and narrowly on a third, but not on the fourth; its other results partly depend on the benchmark's wording. On the Gender-Secret fine-tune it clears the band by a few hundredths on Llama and fails on Mistral. The sum of the two directions ranks lies better than a supervised probe at the same depth on all four models (on one, both near chance) but is level with the persona direction on Llama's average and behind that probe on held-out lies. More varied role-play questions do not help the persona direction on Liars' Bench.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Liars' Bench, Llama-3.3-70B, Qwen, AI safety.

Source attribution: curated from arXiv cs.AI (@Md. Abdullah Al Zami, Suraiya Shanaz, Mahdi Hasan, Shoaib Ahmed Dipu, Farig Yousuf Sadeque, Md. Sabbir Ahmed).