DeoGen：自动生成义务推理基准的论文方法
Deogen: Automatic Benchmark Generation for Reasoning over Policies
做了个叫 DeoGen 的工具，能把政策规则自动变成推理测试题。测出前沿模型该拒答时只有 22-40% 答对，这个盲区挺扎眼的。
arXiv 论文提出 DeoGen，一种可自动生成义务推理（deontic reasoning）基准的工具。它把政策规则自动形式化为一阶逻辑，构建决策树并枚举每条路径作为基准条目，树越深条目数量组合式增长。叶子路径对应可判定的裁决，非叶子路径则测试模型在信息不足时能否拒绝作答。评测显示前沿模型在可判定条目上正确率 89-92%，但在信息不足的条目上即使被明确要求弃权，正确率也只有 22-40%。
Deogen: Automatic Benchmark Generation for Reasoning over Policies
Language models are increasingly being deployed in regulated settings, where a model must apply given rules to the facts of a situation to determine a verdict. Benchmarking this capability, known as deontic reasoning, remains a costly and manual process, with existing datasets largely hand-crafted. In this paper, we present DeoGen, a scalable, automatic, and parametrizable benchmark generator for deontic reasoning. By autoformalizing policy rules to first-order logic, we can construct decision trees that capture possible outcome trajectories. Each path down a tree becomes a benchmark entry, with each tree producing combinatorially many entries relative to tree depth. Paths ending at a leaf entail a verdict, while non-leaf paths measure a model's ability to abstain when presented with insufficient information. Evaluations show frontier models return the correct verdict 89-92% of the time. Notably, performance degrades (22-40% correct) on entries without sufficient information to claim a verdict, even with explicit instructions to abstain in such cases. In all, DeoGen presents an automatic, scalable, and parametric method for generating deontic reasoning benchmarks.