LLM能否测试终端用户界面?

Can LLMs Test Terminal User Interfaces?

精选理由

这篇论文把197个终端界面应用打包成基准,拿4个大模型和随机点击对比,发现随机探索还挺能打,但大模型单次操作效率更高。他们开源了tuicov和tuibot,想测TUI的可以试试。

AI 摘要

论文调研了197个真实TUI应用,发现仅12%的测试代码涉及界面,其中45%从不发送输入。作者将ratatui/Rust、bubbletea/Go、textual/Python、ink/TypeScript应用打包成无头基准,对比4个前沿LLM与随机探索。在相同时间预算下,随机探索是强基线,但每次交互LLM引导更高效,且能独特到达需要输入触发的故障。自动派生启动输入带来最大实际收益,使原本无法启动的应用得以运行。行覆盖率无法有效预测崩溃发现,说明它不宜作为测试有效性的代理指标。

原文 · arXiv cs.AI

Can LLMs Test Terminal User Interfaces?

Terminal User Interfaces (TUIs) combine the stateful, screen-oriented behaviour of GUIs with terminal deployment and are now common in developer tools. Yet they lack a dedicated testing methodology. We survey 197 real-world TUI applications: only 12% of test code exercises the interface, and 45% of those tests never send input, checking a static frame instead. We turn these applications into a headless benchmark spanning ratatui/Rust, bubbletea/Go, textual/Python, and ink/TypeScript, packaging each as an instrumented Docker image. We record line and widget coverage where reliable, rendered terminal states, and crashes. Under equal wall-clock budgets, we compare four frontier LLMs with random exploration. No model dominates. Random is a strong time-budgeted baseline, but its crash advantage comes from higher throughput: per interaction, LLM guidance is more efficient and uniquely reaches input-gated faults. Automatically deriving launch inputs yields the largest practical gain, enabling applications that otherwise never start. Line coverage poorly predicts crash discovery, weakening it as a proxy for test effectiveness. Automated TUI testing is feasible but far from solved, and honest baselines matter more than model choice. We release the coverage tool tuicov at https://github.com/tui-testing/tuicov and the testing framework tuibot at https://github.com/tui-testing/tuibot.