SPARK 提出语义保持提示变异框架提升 LLM 代码生成可靠性
SPARK: Semantic-preserving Prompt Mutation for Code Generation using Consensus-based Verification Framework
做代码生成相关工作的人可以看看这篇。它不改模型本身，只靠对题目做改写、重命名这类小变异再跑共识验证，就在 LiveCodeBench 等三个基准上比现有共识方法高了最多 4.1%，没有测试用例的团队尤其用得上。
SPARK 是一个针对 LLM 代码生成的共识验证增强框架，在缺乏测试用例的真实场景下提升生成代码的可验证性。它在 LiveCodeBench、BigCodeBench 和 DLBench 三个基准上测试了 o4-mini、DeepSeek-R1-Basic 和 Gemini-2.5-Pro 三个模型。
技术上，SPARK 对问题描述施加语义保持变异，包括改写、语法调整、函数重命名等操作，然后对每个变异版本生成一组代码解法和测试用例，再通过共识验证衡量候选解法通过测试的一致性，以此筛选更稳健、更正确的代码。
实验结果显示，SPARK 相比现有最先进的共识验证方法最高提升 4.1%。泛化实验表明，在传统验证资源稀缺的场景下，这种变异策略可以有效增强基于 LLM 的代码生成系统的可靠性，定性分析也解释了变异为何能引导模型产出更好的解法。
SPARK: Semantic-preserving Prompt Mutation for Code Generation using Consensus-based Verification Framework
Automated code generation with large language models (LLMs) has improved in recent years. With the lack of test cases in real-world usages, generated code is not automatically verifiable and often requires manual validation. Previous studies have leveraged consensus verification, where code solutions and corresponding test cases are generated and evaluated based on functional consistency. Building on this, SPARK incorporates semantic-preserving mutations to problem descriptions such as paraphrasing, grammatical adjustments, function renaming, and other alterations to enhance the robustness and correctness of LLM-generated code. For each mutated description, SPARK generates a set of code solutions and test cases. These candidate solutions are evaluated using consensus-based verification, where the solutions' quality is measured by how consistently they pass generated tests. We assess the effectiveness of SPARK on three benchmarks (LiveCodeBench, BigCodeBench, and DLBench) across three LLMs (o4-mini, DeepSeek-R1-Basic, and Gemini-2.5-Pro). Our evaluation demonstrates that SPARK outperforms state-of-the-art consensus-based verification approaches by up to 4.1%. Our generalization experiment highlights the practical potential of SPARK as an effective strategy for enhancing the reliability and correctness of LLM-based code generation systems in scenarios where traditional validation resources are scarce, while our qualitative analysis shows insight into how SPARK's mutation helps LLMs generate better code solutions.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include SPARK, LiveCodeBench, Gemini-2.5-Pro, DeepSeek.
Primary source verification: published directly by arXiv: DeepSeek with first-party authority.