指令文件对AI智能体PR的影响:Instructions-as-Code研究

Toward Instructions-as-Code: Understanding the Impact of Instruction Files on Agentic Pull Requests

精选理由

做AI辅助开发的团队会发现,指令文件不是写得越多越好——研究揭示了哪些写法真正有效,建议点开看看如何优化你的项目指令。

AI 摘要

该研究分析了148个项目中15549个由AI智能体(如GitHub Copilot)生成的拉取请求(Agentic-PRs),探讨指令文件对AI智能体性能的影响。研究发现,创建指令文件并不总是提升合并率:27.7%的项目合并率提升至少20%,但26.35%的项目反而下降。代码变更量和合并所需时间等指标也呈现类似分化。初步探索表明,成功提升合并率的项目拥有更长、结构更清晰的指令文件。研究呼吁将指令文件开发视为软件工程活动(Instructions-as-Code),以帮助实践者优化AI协作。

原文 · arXiv cs.AI

Toward Instructions-as-Code: Understanding the Impact of Instruction Files on Agentic Pull Requests

AI-agents (e.g., GitHub Copilot) collaborate as teammates in different software engineering tasks, including code generation proposed through pull requests (Agentic-PRs). For better agent efficiency, developers create instruction files that guide the AI-agents, including how to navigate the project, locate the right components, run tests, respect best practices, and more. In this paper, we investigate the relationship between the creation of these instructions and the performance of AI-agents in creating better pull requests, which have a higher chance of success (i.e., the merge rate), address more complex tasks (e.g., code churn), and require less effort to be merged (e.g., time to merge). To this end, we analyze 15,549 agentic PRs from 148 projects in the AIDev dataset. Using the three dimensions, we compare each project before and after the creation of the instruction files. We find that specifying instructions for AI-agents does not necessarily lead to better results. With the instruction files, 27.7\% of the projects increased their merge rate by at least 20\%, while 26.35\% decreased it. The same observation is seen with the amount of changes (e.g., code churn, number of modified files) and with the efforts to merge an agentic PR (e.g., merge time and number of comments). From a first exploration, we find that projects that managed to increase their merge rate have substantially longer instruction files, which are also well structured into a higher number of sections and sub-sections. Our results motivate the need for research to assist practitioners in framing the development of instruction files as a software engineering activity (aka, \textbf{Instructions-as-Code}).