研究发现 93.8% 的 LLM 生成重构候选在仓库层面属高契约/依赖风险
When LLM-Generated Refactorings Ripple: An Empirical Study of Repository Adoption Readiness
写代码用 LLM 重构的朋友看看：1500 次实验里 93.8% 的重构在仓库层面有高风险，单提示和多智能体还没差别。
一项实证研究在 15 个开源 Java 仓库的 150 个生产文件上进行 1,500 次重构生成尝试，涵盖五种 LLM 配置和单提示、LangGraph 多智能体两种生成方式。结果显示 1,365 个候选（93.8%）被归为高契约/依赖风险，290 个（19.9%）存在假的行为等价指标，只有 8 个满足低风险标准。研究还发现单提示与 LangGraph 多智能体在主要风险指标上无统计学显著差异。作者建议将 LLM 生成重构视为需要分阶段仓库级验证的候选方案。
When LLM-Generated Refactorings Ripple: An Empirical Study of Repository Adoption Readiness
Large Language Models (LLMs) are increasingly used to refactor source files, but file-local evaluations do not reveal changes to contracts, dependencies, or caller assumptions affecting repository adoption. To address this gap, we conduct 1,500 generation attempts across 150 production files from 15 open-source Java repositories, five LLM configurations, and two dependency-aware generation conditions: single-prompt refactoring and LangGraph-based multi-agent refactoring. Of these attempts, 1,455 yield complete candidates eligible for analysis. We evaluate these candidates using local changes, repository-facing structural indicators, detected caller impact, the behavioural-equivalence indicator, and available test evidence, and categorize them as Low-risk, Review-needed, High Contract/Dependency Risk, or Inconclusive. The results show that 1,365 candidates (93.8%) are classified as High Contract/Dependency Risk, whereas 106 (7.3%) have at least one detected broken downstream caller and 290 (19.9%) have a false behavioural-equivalence indicator; only eight satisfy the Low-risk criteria. Within the reason-covered subset, the High-risk rate decreases under more conservative dependency-delta thresholds, while 60 of the 65 candidates with conclusive targeted-test outcomes pass, including 50 classified as High risk. Paired analyses of 705 complete pairs identify no statistically significant generation-condition difference in the principal risk indicators, although model configurations differ in specific detected risk channels. These findings suggest that practitioners should treat LLM-generated refactorings as candidates requiring staged repository-level verification, while future research should validate screening and extraction precision and strengthen compilation, behavioural, and coordinated multi-file verification.
