这篇论文搞了一个对抗性语用学基准，有18个具体条目，专门测试模型被指令冲突、嵌入命令搞昏头的情况，比以前只打通过/不通过标签靠谱多了。
该论文提出对抗性语用学作为评估语言模型安全性的基准与标注协议，涵盖指令冲突、嵌入命令、引用、范围歧义、指示语、间接言语行为和多轮智能体转录。论文贡献包括一个18条目的种子基准（含验证者强制元数据）和一个54行的本地种子试点实验。专家评估协议区分任务成功、政策合规、安全风险、拒绝结果和评估者置信度。框架为验证安全评估、LLM裁判、黄金集构建、提示注入测试和安全文档提供了实用工具。
Adversarial Pragmatics for AI Safety Evaluation: A Benchmark for Instruction Conflict, Embedded Commands, and Policy Ambiguity
Safety evaluations for language models increasingly depend on judgments about ambiguous natural-language behaviour: whether a model has followed an instruction, refused appropriately, complied with a policy, resisted an embedded command, or misreported progress in an agentic task. Existing benchmarks often compress these distinctions into pass/fail labels, obscuring whether failures arise from capability limits, policy ambiguity, instruction conflict, scaffold failure, or unstable evaluator judgments. This paper introduces adversarial pragmatics as a benchmark and annotation protocol for evaluating model behaviour under instruction conflict, embedded commands, quotation, scope ambiguity, deixis, indirect speech acts, and multi-turn agent transcripts. The contribution is empirical and methodological: a linguistically controlled taxonomy, an 18-item seed benchmark with validator-enforced metadata, a 54-row local seed pilot, an expert-evaluation protocol distinguishing task success, policy compliance, safety risk, refusal outcome, and evaluator confidence, and metrics for judge validity, diagnostic ambiguity, and taxonomy drift. The framework turns linguistic judgment methodology into a practical tool for validating safety evals, LLM judges, gold-set construction, prompt-injection tests, and safety documentation.