想测测视觉语言模型能不能只看截图就生成可交互的网页应用？这篇论文提出了UI2App基准，发现模型在视觉还原上还行，但交互推理差得远，最高分才7.5。
论文提出UI2App基准，专门评估从截图推断交互行为的能力，无需文本或行为指导。基准包含327张截图，分为45个状态一致组，用于多路由网页应用。评估沿四个维度：可执行性、导航可达性、视觉保真度和交互推理，其中交互指标(IIS)按功能正确性和状态管理复杂度评分。在6个前沿视觉语言模型上，视觉保真度领先者仅得7.5的IIS，落后IIS冠军5.2倍；半数模型在高复杂度跨页面状态交互上得零。结果显示从静态截图推断完整交互行为仍是关键挑战。
UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation
Large language models (LLMs) have demonstrated growing competence in web page generation. However, existing text-driven approaches rely on complex prompts that impose substantial demands on users and offer limited expressivity for page layout and cross-page visual coherence. Image-driven paradigms, which take UI screenshots as input, align more closely with real development workflows. However, current benchmarks focus primarily on visual fidelity and lack a systematic evaluation of the interaction capabilities in generated artifacts. To address this gap, we introduce UI2App, the first benchmark targeting interaction inference, the ability to recover application behavior from screenshots alone, without any textual or behavioral guidance. UI2App comprises 327 screenshots grouped into 45 state-coherent screenshot sets for runnable multi-route web applications. We design an end-to-end pipeline that evaluates each artifact along four dimensions: executability, navigation reachability, visual fidelity, and interaction inference. The interaction metric (IIS) assesses inferred interactions by functional correctness and state-management complexity, crediting any valid implementation rather than matching a single reference. Experiments on six frontier vision-language models reveal a marked capability mismatch between visual reconstruction and interaction realization: the visual-fidelity leader scores only 7.5 on IIS, ranking fourth and trailing the IIS leader by 5.2x. High-complexity interactions such as cross-page state remain a pervasive bottleneck, with half of the evaluated models scoring exactly zero on this dimension. Overall, the results indicate that inferring complete interaction behavior from static screenshots remains a key challenge for models.