Verify the Verifier:用 LLM 交叉检查 UVM 记分板漏掉的规格级 Bug
Verify the Verifier: Specification-Grounded LLM Cross-Checking for Bugs That Escape UVM Scoreboards
芯片验证团队值得看:UVM 回归全绿不等于没 bug。论文在四个 OpenTitan IP 上插入 564 个 bug,用 LLM 对照规格文档审查 trace,确认了 41 个记分板漏掉的违规,方法可直接套到自家 testbench。
一篇 arXiv 论文提出 Verify the Verifier 框架,用基于 LLM 的 Anomaly Cross-Checker 对照规格说明,审查已通过的 UVM 回归测试 trace,捕捉记分板(scoreboard)漏掉的 bug。作者将这类通过全部测试却违反规格的 bug 称为 scoreboard escapes。在四个 OpenTitan IP 中插入 564 个 bug 后,框架识别出 68 个候选 scoreboard escapes,最终确认其中 41 个:这些 bug 在带 bug 设计上触发断言失败,并能在原设计上证明通过。其机制是让 LLM 将实际执行 trace 与规格条款逐项比对,弥补记分板只验证与参考模型等价的盲区。
这回答了硬件验证的老问题:UVM 回归全绿只代表设计与参考模型在检查点上一致,不代表满足规格。以往靠人工补充断言,现在由 LLM 跨检查生成规格级断言并交叉验证。对使用 OpenTitan 等开源 IP 或维护成熟 testbench 的验证团队,论文给出了可直接复用的流程与量化结果。
Verify the Verifier: Specification-Grounded LLM Cross-Checking for Bugs That Escape UVM Scoreboards
A passing UVM regression shows the design agrees with the reference model on what the scoreboard checks, not that it meets the specification. Bugs that pass every test yet violate the specification, termed scoreboard escapes, survive mature testbenches. We present Verify the Verifier, to our knowledge the first framework whose LLM-based Anomaly Cross-Checker examines a passing regression's traces against the specification. Of 564 bugs inserted into four OpenTitan IPs, 68 are candidate scoreboard escapes: non-equivalent, exercised by the tests, yet passing. The framework confirmed 41 as scoreboard escapes with assertions failing on the buggy design and proven on the original.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include LLM, UVM, OpenTitan, chip verification.
Source attribution: curated from arXiv cs.AI (@Ana E. Sanchez Villalobos, Michael D. Quinn, Aakash Tyagi, Minh D. Luu, David Kebo Houngninou).