GUI-JEPA：无解码器架构让移动端 GUI 智能体提速 4 倍
GUI-JEPA: Candidate-Conditioned JEPA Latents as a Fast Action Policy for Mobile GUI Agents
上海交大做了个不走 VLM 逐字解码的 GUI 智能体，比 Qwen2.5-VL-7B 快 4 倍、参数只要 6%，做移动端自动化的可以看看。
上海交大团队提出 GUI-JEPA，首个基于 JEPA 的无解码器 GUI 智能体策略，直接在潜空间中选动作，不逐 token 生成。模型用未标注 GUI 截图预训练编码器，动作与区域条件的潜空间预测器把屏幕表征转为每个候选动作的潜向量，轻量头一次批量打分。在 AndroidControl 基准上，GUI-JEPA 比同输入微调的 Qwen2.5-VL-7B 智能体快 2.8-4.0 倍，达到其 86-92% 的分布内准确率，参数量仅 6%。实验还发现预测器随机初始化效果与预训练版本相当，说明它学的是决策本身而非屏幕动态。代码已在 GitHub 开源。
GUI-JEPA: Candidate-Conditioned JEPA Latents as a Fast Action Policy for Mobile GUI Agents
Mobile GUI agents are typically fine-tuned 7B-scale vision-language models (VLMs) that generate each action as text. However, these VLMs decode every action token by token, and even agents that score candidate actions keep a large language model or VLM in the loop, making them slow and memory-hungry. To address these issues, we propose GUI-JEPA, to our knowledge the first decoder-free GUI agent policy built on a Joint-Embedding Predictive Architecture (JEPA), which selects actions in the latent space of a JEPA encoder pretrained on unlabelled GUI screenshots. Its action- and region-conditioned latent predictor turns the screen latent into one latent per candidate action, letting lightweight heads score all candidates against the instruction in a single batched pass, without decoding a token. We further show that the scorer relies on this latent pathway and that the predictor is trained for the decision itself: a randomly initialised predictor does just as well as one pretrained on screen dynamics. On AndroidControl, GUI-JEPA is 2.8-4.0x faster on the same GPU than a Qwen2.5-VL-7B agent we fine-tune with the same inputs, while reaching 86-92% of its in-distribution accuracy with 6% of its parameters. Code is available at https://github.com/bcwang-sjtu/GUI-JEPA
