新基准揭示现有谬误检测数据集存在缺陷
Fallacy Benchmarks Measure Scheme Recognition, Not Fallacy Detection
这篇论文指出当前用于衡量AI检测谬误能力的基准存在严重缺陷，可能会误导评估结果，对研究者和开发者都有参考价值。
现有谬误检测基准将谬误类别与一个单一的“有效”或“无”类别配对，这种构建方式具有误导性。一个分类器可以通过学习数据收集未标记为谬误的线索来表现良好，而无需学习区分谬误和正确论证。我们表明，基准报告中报告的低误报率是这种类别构建方式的结果，而非检测能力的证据。对于谬误，最有信息量的负面样本是使用相同论证模式的正确论证，在我们分析的四个基准中，这种论证在“有效”类别中最多占百分之几。在构建的方案匹配负面样本上评估时，CoCoLoFa的误报率从16.6%上升到58.9%，Reddit从5.7%上升到62.0%。该率取决于负面样本的写法，因此我们比较了来自同一管道的两个条件，它们仅方案身份不同。分类器将方案匹配的负面样本标记为源谬误类型的概率比错误方案负面样本高40.9个百分点，后者85.9%的时间被识别为实际使用的方案，而源类型仅0.4%。分类器学习的是论证使用的方案，而非是否正确使用，并且在基准自身的测试集上，这两者是无法区分的。在从未见过这些基准的三个零样本LLM检测器中也出现了相同的解耦现象，在故意构建的负面类别上，测量值要低得多。我们发布了这些项目作为Scheme Foils。直到“有效”类别经过方案匹配覆盖审计后，报告的误报率才应被视为检测能力的衡量指标。
Fallacy Benchmarks Measure Scheme Recognition, Not Fallacy Detection
Fallacy-detection benchmarks pair fallacy classes with a single "valid" or "none" class that takes everything data collection did not label as a fallacy. This construction is misleading: a classifier can learn cues that do well on this class without learning to tell a fallacy from a correct argument. We show that the low false-positive rates benchmarks report are an artifact of how the class is built, not evidence of detection ability. The most informative negative for a fallacy is a correct argument using the same argumentation scheme, and such arguments are at most a few percent of the valid class across the four benchmarks we examined. Evaluated on constructed scheme-matched negatives, false-positive rates rise from 16.6% to 58.9% on CoCoLoFa and from 5.7% to 62.0% on Reddit. That rate depends on how the negatives are written, so we also compare two conditions from the same pipeline that differ only in scheme identity. Classifiers label scheme-matched negatives as the source fallacy type 40.9 points more often than wrong-scheme negatives, which are instead identified as the scheme they actually use 85.9% of the time against 0.4% for the source type. The classifier has learned which scheme an argument uses, not whether it uses it correctly, and on the benchmarks' own test sets the two are indistinguishable. The same dissociation appears in three zero-shot LLM detectors that never saw these benchmarks, and the measurement is far lower on a negative class that was built deliberately. We release the items as Scheme Foils. A reported false-positive rate should not be trusted as a measure of detection until the valid class has been audited for scheme-matched coverage.