做 AI 编码智能体或进化算法研究的开发者，这篇论文帮你拆解了 benchmark 分数的真实来源——别再只看最终得分了，EvoTrace 让你看清智能体到底在“进化”什么。
最新研究指出，将 LLM 与进化搜索结合的编码智能体在数学发现和算法设计上表现强劲，但进步可能源于多种机制：新算法结构、调整现有策略、重组模型内部知识或过拟合评估器。为区分这些机制，研究者推出了 EvoTrace 数据集，涵盖四种进化框架、推理与非推理模型及 16 个任务。他们开发了 EvoReplay 方法，通过重放分析高分解背后的局部搜索状态，并标注了九种编辑类型。结果发现，大部分得分提升来自少数编辑类型，且约 30% 的代码行是重复引入的已删除行，表明基准进步可能并非真正的新算法结构。
What Do Evolutionary Coding Agents Evolve?
Recent work pairs LLMs with evolutionary search to iteratively generate, modify, and select code using task-specific feedback. These systems have produced strong results in mathematical discovery and algorithm design, yet a fundamental question remains: what do they actually evolve? Progress is typically summarized by the best score a run reaches under a task-specific evaluator, but that score can reflect several different mechanisms: new algorithmic structure, re-tuning an existing strategy, recombining ideas already in the model's internal knowledge, or overfitting to the evaluator. Distinguishing these mechanisms requires inspecting the search process itself, not only its final outcome. We introduce EvoTrace, a dataset of evolutionary coding traces spanning four evolutionary frameworks, reasoning and non-reasoning models, and 16 tasks across mathematics and algorithm design. To analyze these traces, we develop EvoReplay, a replay-based methodology that reconstructs the local search states behind high-scoring solutions and tests controlled interventions, including adjusting constants, removing program components and substituting models or prompting contexts. We annotate every code edit in EvoTrace with one of nine recurring edit types using an LLM-as-judge pipeline validated against blind human re-annotation. Across EvoTrace, most score gains come from a small subset of these edit types. We further find a deterministic cycling pattern: about 30% of code lines added during search are byte-identical re-introductions of previously-deleted lines, present throughout nearly every run. These results show that benchmark gains in evolutionary coding agents can arise from qualitatively different mechanisms, only some of which correspond to new algorithmic structure. EvoTrace enables more diagnostic evaluation of evolutionary coding agents beyond final benchmark scores.