可解释AI对AI辅助代码审查中信任的影响评估

Evaluating the Impact of Explainable AI on Trust in AI-Assisted Code Review

精选理由

这篇论文告诉你:给AI代码审查加解释,反而可能让人更不听话。详细解释信任高但同意率低,这结论挺反直觉的。

AI 摘要

一篇针对34名程序员的用户研究发现,在AI代码审查中提供详细解释(条件A)获得最高信任评分(3.99/5),但仅提供审查反馈(条件B)反而获得最高同意率(89.22%),说明更多解释会促使开发者更频繁地质疑AI建议。无解释(条件C)的信任和同意率均最低。解释程度并未显著影响审查耗时。该研究为设计可信的AI代码审查系统提供了实证依据。

原文 · arXiv cs.AI

Evaluating the Impact of Explainable AI on Trust in AI-Assisted Code Review

Background: Large language models (LLMs) are increasingly used to automate code review, but the reasoning behind their decisions remains hard to understand. Developers struggle to assess the validity of LLM-generated reviews, making it difficult to gauge how much trust to place in them. The role of Explainable AI (XAI) in code review and its impact on trust remain underexplored. Objective: We study the influence of XAI on developer trust in AI-assisted code reviews. Method: We conducted a within-subjects user study with 34 participants, comparing three LLM-based code review systems with varying levels of XAI support: Condition A (detailed explanation and review feedback), Condition B (review feedback only), and Condition C (no explanations). Participants reviewed real-world code change requests alongside the AI-generated reviews. We measured trust perceptions, agreement with the AI recommendation, the reasoning given for each decision, and the time taken. Results: The level of explanation significantly influences both trust and agreement with AI recommendations, but in different ways. Full explanations (A) yield the highest perceived trust (M = 3.99/5) but not the highest agreement, whereas moderate explanations (B) achieve the highest agreement (89.22%). This could suggest that more explanation prompts developers to question AI recommendations more frequently. No explanations (C) results in the lowest trust and agreement. Explanation level did not significantly affect review time. The most commonly cited reasons for decisions were code readability and correctness. Conclusion: Incorporating XAI into code review significantly changes trust perceptions and agreement with AI recommendations. These results inform the design and evaluation of trustworthy AI-based code review systems, as well as studies on the human factors of AI-assisted software development.