软件工程团队和AI研究者终于有了一个严谨的测试套件质量评估工具——SWE-Mutation能帮你判断LLM生成的测试是否真的可靠，做自动化测试或代码修复的开发者值得关注。
SWE-Mutation 是一个新基准，用于评估大语言模型（LLM）生成的测试套件的质量。它通过引入系统性的变异解决方案来“欺骗”测试套件，从而衡量测试套件的判别能力。该基准包含从800个原始实例衍生的2,636个变异变体，并覆盖九种编程语言。实验表明，即使是DeepSeek-V3.1，其验证率也仅为10.20%，检测率为36.15%，暴露了当前LLM在生成可靠测试套件方面的严重不足。该研究还提出了一种智能体驱动的变异策略，使测试套件更难被欺骗，从而更真实地反映LLM的能力缺陷。
SWE-Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering?
Evaluating software engineering capabilities has become a core component of modern large language models (LLMs); however, the key bottleneck hindering further scaling lies not in the scarcity of high-quality solutions, but in the lack of high-quality test suites. Test suites are indispensable both for synthesizing program repair trajectories and for providing precise feedback signals in reinforcement learning. Unfortunately, due to the high cost and difficulty of annotation, high-quality test suites have long been hard to obtain, while those automatically generated by LLMs tend to be superficial and lack sufficient discriminative power. As a first step toward constructing high-quality test suites, we introduce SWE-Mutation, a benchmark for evaluating LLM-generated test suites. The benchmark characterizes test suites by introducing systematically mutated solutions that attempt to ``fool'' the test suites and pass validation. We further propose an agentic, language-agnostic framework for automatically generating complex mutants. Our benchmark consists of 2,636 mutated variants derived from 800 original instances and includes a multilingual subset spanning nine programming languages. Experiments on seven LLMs reveal that even DeepSeek-V3.1 achieves only 10.20% verification and 36.15% detection rates, highlighting the inadequacy of current LLMs. Additionally, our agentic mutation strategy enhances realism, reducing average detection rates from 71.04% to 39.81% compared to conventional methods. These findings expose persistent deficiencies in the ability of current LLMs to generate reliable and discriminative test suites.