研究：8B/9B 评分模型会把越狱结果误归为规避类，导致攻击成功率相差 22 倍
The Missing Compliance Class: LLM Graders That Collapse Jailbreak Success, and a Free Check
同一批越狱结果换个打分模型，成功率能差 22 倍，这论文告诉你怎么免费检查自己的评测有没有踩坑。
这篇 arXiv 论文固定目标模型、提示词和响应，只更换评分模型，考察越狱攻击成功率如何变化。来自三个模型家族的 8B 和 9B 评分模型会把本应判为 compliance 的响应归到相邻的 evasion 类，其中一个在 900 条响应中将 compliance 标签赋值为零次。在 900 条响应、8 个评分模型的两组语料上，compliance 比例最高相差 22 到 25 倍，non-refusal 比例相差 18 到 23 个百分点。论文提出报告最低标准：公布 per-class 标签计数和所用评分模型的名称，并同时报告 non-refusal 比例。
The Missing Compliance Class: LLM Graders That Collapse Jailbreak Success, and a Free Check
Jailbreak papers report an attack success rate, the share of responses that deliver the harmful request, and read it as a property of the target model. We hold the target model, the prompts and the responses fixed and change only the grader. Three dense graders of 8B and 9B parameters, from three model families, move compliant responses to the neighbouring class, evasion, instead of scattering them across classes. One of them assigns the compliance label zero times in 900 responses. This class collapse also appears on new harmful requests, on a target model from a third family, and in a held-out 7B grader from a fourth family. Per-class label counts expose it at no cost and without human labels. The collapse decides how the rate behaves under a change of grader. On two corpora of 900 responses and eight graders, the compliance rate differs by a factor of 22 to 25, while the non-refusal rate differs by 18 to 23 percentage points. Without the three collapsing graders, compliance differs by a factor of 1.7 to 2.8 on all four corpora, and non-refusal still differs by 10 to 22 points. These results do not change when every grader is called once. We propose a reporting minimum that keeps the compliance rate and adds its per-class label counts and the named grader. Beside it goes the non-refusal rate, whose disagreement across graders is additive.
