AI审稿能否提升论文写作?20篇计算机架构论文实证研究

Can AI Review Improve Paper Drafting? An Empirical Study on 20 Computer Architecture Submissions

精选理由

做计算机架构研究或论文写作的学者,可以看看AI审稿如何帮你发现盲点——工具已开源,值得一试。

AI 摘要

该研究通过20篇计算机架构论文的案例,探讨AI审稿与人类审稿的一致性。研究者开发了AI-Paper-Review工具,可生成结构化AI审稿意见,并与人类审稿意见对比。结果显示AI能覆盖大部分人类提出的问题,还能发现人类遗漏的缺陷。研究旨在探索AI辅助论文写作的潜力,而非鼓励替代同行评审。工具和数据已开源,供后续研究使用。

原文 · arXiv cs.AI

Can AI Review Improve Paper Drafting? An Empirical Study on 20 Computer Architecture Submissions

Research is advancing faster than ever with artificial intelligence (AI); and so are the corresponding research papers. The exploding volume of AI-generated papers have put a strain to peer review, leading to the usage of AI-generated review, potentially wide yet sneaky. However, relevant ethical concerns about confidentiality, quality, and fairness are raised and no consensus has been reached in the broad research community. We expect the debate to continue for a while, but in the meantime, we ask an alternative, practical question: \textit{can AI review improve paper drafting?} We study 20 computer architecture papers, with varying levels of submission lineage, to expose how well AI review aligns with human review, quantified by a set of metrics we define. To conduct the case study, we build a web UI-integrated tool, \emph{AI-Paper-Review}, that generates structured AI review of a draft paper, available at https://github.com/unarylab/ai-paper-review. This tool selects several AI reviewers from a diverse pool of AI reviewers and clusters and ranks their comments based on commonality and importance of review comments. It also allows to align AI comments with human comments to facilitate metric-based validation. The case study shows that AI review can cover a significant fraction of human-raised issues, but also raises issues missing in human review. This paper is not intended to encourage using AI for peer review at the current stage, but to study that (1) how AI review can improve paper drafting and (2) the potential and limitation of AI-based peer review. The release of the tool and the case study data is intended to instigate future research on this topic. Misuse for peer review would violate the ethics policies from major academic venues.