V-JEPA-LM:让视频语言模型真正读懂动态信息
V-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations
如果你在做视频理解或智能体,这篇值得花半小时。它证明了 V-JEPA 这类世界模型编码器里的动态信息用反事实监督就能读出来,4B 模型比开源 8B 还高 7 分,训练只要一个阶段和 4.53M 样本,成本低很多。做视频问答、时序推理的团队可以直接拿开源代码复现。
arXiv 论文 V-JEPA-LM 解决视频语言模型(VLM)读不出动态信息的问题:冻结的 V-JEPA 特征用线性探针可以从同一批样本中读出速度(87% 准确率),但接上语言模型后 VLM 的速度问答只有随机水平,说明编码器携带的动态信息在训练中被浪费。论文定位到动态信息存在于 V-JEPA 这类预测式编码器跨视频共享的子空间中,而 SigLIP 这类逐帧、文本对齐的编码器没有这一共享成分。
团队用像素变换构造反事实监督:把公开视频变成答案平衡的变体,答案由变换本身决定,模型只能通过读出变化来答对。模型(1B 到 8B,基于 Qwen3 和 Llama)仅在单阶段用 4.53M 公开样本、每帧 32 个视觉 token 训练,而现有开源视频 LLM 通常需要多阶段、9M 到 172M 样本和 64 到 256 token。结果 4B 版本在十一个视频基准的平均分达 72.1,超过最好的开源 4B(61.1)和 8B(64.8)。控制变量实验表明编码器决定能读什么、监督决定读什么:同等预算下反事实监督平均提升 4.5 分,自然基准的速度类问题上提升 38 分。代码和模型已在 GitHub 开源。
V-JEPA-LM: Video-Language Models That Read Dynamics Beyond Static, Text-Aligned Representations
World-model video encoders such as V-JEPA learn how scenes change (direction, speed, order, state) from video alone. To answer questions, they are joined to a language model trained on captions and question-answer pairs. But each caption or question names only a few attributes of a clip, so the model learns to retrieve those attributes rather than to read the clip, and the dynamics the encoder carries go unused: on the same items, a linear probe reads speed from frozen V-JEPA features at 87% while the VLM answers at chance. We show where dynamics lives in a frozen encoder: in a subspace shared across videos for predictive encoders such as V-JEPA, but not as a shared component of frame-wise, text-aligned encoders such as SigLIP. We then build counterfactual supervision that makes the language model read it: pixel transformations turn public videos into answer-balanced variants whose answers are set by the transformation and differ only in the change. Trained in a single stage on 4.53M public samples with 32 visual tokens per frame, where open video LLMs use several stages, 9M to 172M samples and 64 to 256 tokens per frame on average, the resulting V-JEPA-LM models (1B to 8B, Qwen3 and Llama) lead every size class in average score across eleven video benchmarks; our 4B model scores 72.1, against 61.1 for the best open 4B and 64.8 for the best open 8B model. A controlled encoder x supervision study shows that the encoder decides what can be read and the supervision what is read: at a matched budget, counterfactual supervision adds 4.5 points on average and 38 on the speed questions of natural benchmarks. Code and models: https://github.com/kkt94/Llama-V-JEPA-LM.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include V-JEPA, Qwen3, Llama, video understanding.
Source attribution: curated from arXiv cs.AI (@Keuntae Kim, Dongjun Kim, Yong Suk Choi).