做代码基准测试的朋友可以看看，PAIChecker能自动揪出PR和Issue配错的情况，SWE-bench上准确率超九成。
PAIChecker是一个多智能体系统，用于检测SWE-bench类基准中PR与Issue配对不对齐的问题。研究者在SWE-bench Verified实例中发现13.6%存在不对齐，涵盖五种模式和十一个细粒度场景。PAIChecker采用三阶段设计，结合特定模式识别、跨智能体标签合成和代码级验证。在SWE-Gym和SWE-bench Multilingual上，该系统分别达到92.12%和91.67%的二元准确率，优于四种LLM后端。
PAIChecker: Uncovering and Checking PR-Issue Misalignment in SWE-Bench-Like Benchmarks
SWE-bench-like benchmarks are widely used for evaluating LLM's issue resolution capability. They typically follow a common construction pipeline: each PR (Pull Request) is paired with its linked issue by extracting issue references from the PR description; the issue description is used as the problem statement, and the PR patch serves as the test oracle. However, due to the inherent complexity of developing and maintaining large repositories, such PR-Issue pairings are often misaligned in practice. In this work, we systematically study SWE-bench Verified instances, finding that 13.6% exhibit misalignment across five patterns in eleven fine-grained scenarios. To enable reliable and scalable construction of those benchmarks in the future, we propose PAIChecker, a multi-agent system for checking PR-Issue misalignment in SWE-bench-like benchmarks. Specifically, PAIChecker adopts a three-phase design that combines specific pattern identification, cross-agent label synthesis, and code-level validation, thereby enabling more accurate, generalizable, and progressively verified detection. Experiments on SWE-Gym and SWE-bench Multilingual show that PAIchecker achieves the best performance across all four LLM backbones, reaching up to 92.12% and 91.67% binary accuracy, respectively.