这篇论文设计了一个人体实验，对比开发者用LLM辅助和手动修补漏洞的速度与安全性，还用了隐藏测试验真假补丁。
一项针对LLM辅助漏洞修补的人体实验研究提出，LLM工具在检测、定位和修复漏洞方面有潜力，但可能引入幻觉或不安全代码。研究计划使用平衡交叉设计，开发了集成隐藏Ghost测试的WebApp，用于验证补丁在功能测试和安全测试下的完整性。试点实验已进行，为后续大规模实验提供初步见解。
Helpful or Harmful? Evaluating LLM-Assisted Vulnerability Patching via a Human Study
Software vulnerability remediation is a cognitively demanding task that requires specialized security expertise often lacking in general developers. In the meantime, Large Language Models (LLMs) assisted tools show potential in vulnerability detection, location, and repair tasks. [Hypothesis:] While LLM-assistance is hypothesized to accelerate patching, it also risks introducing hallucinations or insecure code, leading to a higher likelihood of generating superficial repairs that bypass the standard functionality checks but fail the security validation. [Objective:] We aim to present an empirical experiment, unveiling the capability of LLM-assisted vulnerability patching compared to manual debugging on human participants in real-world scenarios. [Method:] We plan to conduct a controlled experiment using a Balanced Crossover design. For that, we have developed a WebApp for code execution and integrated hidden Ghost Tests to verify patch integrity beyond visible functional requirements. The experiment involves training and evaluation scenarios. The remediation speed, remediation efficacy for both standard functionality tests and security tests, and participant perception will be evaluated. [Pilot Study:] A pilot experiment with a small sample of participants has been conducted, providing insights for the following study.