这篇论文让你知道,给机器人一个语言指令,它就能自己学会怎么转动相机去看你想看的地方——全靠学人类第一人称视频,不用手动标数据。
该论文提出了一个名为LIME的视觉语言相机运动生成器。它根据当前RGB图像和自然语言意图,预测下一视角的相对目标相机位姿(SE(3))。为了训练模型,作者从第一人称视频中挖掘多意图相机运动监督信号,配对合理的意图和观察增益描述。LIME结合了自回归的观察增益输出和连续流匹配位姿头,能够联合预测下一视图该展示什么并代表多假设目标视图。实验表明,LIME可从被动的人类视频中学习主动选择相机位姿,用于下游机器人任务。
LIME: Learning Intent-aware Camera Motion from Egocentric Video
Autonomous robots often need to move their camera before they can act: to inspect an object, reveal an occluded region, or obtain a view that responds to a user's intent. While vision-language navigation translates instructions to base motion and vision-language-action policies map instructions to manipulation actions, language-conditioned camera motion remains comparatively underexplored as a first-class action. We formulate language-conditioned camera motion generation: given a current RGB observation and a free-form natural-language intent, predict a relative target camera pose for the next observation. This task is inherently non-trivial: viewpoint changes are driven by latent perceptual intentions, and a valid motion may operate at different semantic granularity, from entering a room to looking around a corner, inspecting a visible object, or revealing an occluded detail. To model this structure, we mine multi-intention camera-motion supervision from egocentric video, pairing plausible intents and observation-gain descriptions with relative SE(3) target poses. We propose LIME, a vision-language camera-motion generator that combines an auto-regressive observation-gain output with a continuous flow-matching pose head. This design lets the model jointly predict what the next view should reveal while representing multi-hypothesis target views. Across experiments and downstream robotic tasks, we show that LIME can learn to actively choose camera poses from passive human video, turning ordinary egocentric recordings into supervision for intent-aware active perception.