Papers

LynnReal-Hand:用手部动作直接操控的交互式视频世界模型

LynnReal-Hand: Unlocking World Interaction with Hand Control

Why we picked it

用手就能直接操控视频世界模型,不再只靠文字和镜头指令。单张 H200 跑到 18 FPS、延迟 741ms,还能实时交互。做具身智能、交互仿真或者世界模型研究的开发者值得关注,代码和数据都会开源。

一篇 arXiv 论文提出 LynnReal-Hand,一个通过手部控制直接与视频世界模型交互的系统。团队整理了 100K 条覆盖真实与合成环境的手部交互片段,并设计手部感知监督来提升细粒度动作与物体交互的建模质量。

Body

技术上有三个关键组件:用因果流映射实现少步生成,用区域感知分数蒸馏融合视频与图像先验来优化交互动态,并推出轻量视频解码器 Ultra-Fast-VAE 降低推理开销。在 HandBench 上,完整序列模型取得 FVD 181.67、MANO Dice 93.02,三步因果模型的 MANO Dice 也达到 92.49。

实时性能方面,单张 H200 上 540p 流式生成达到 18.03 FPS,延迟 741ms,峰值显存仅 34.89GB。团队承诺开源数据集、模型权重与完整训练推理代码,可支撑手控世界模型方向的后续研究。

Original · arXiv cs.AI

LynnReal-Hand: Unlocking World Interaction with Hand Control

Interactive video world models enable exploration through text and camera controls, yet direct object interaction requires finer control. Hands provide a natural interface, but translating continuous hand movements into coherent visual responses remains challenging because of complex hand-object dynamics and demanding real-time computation. To address these challenges, we introduce LynnReal-Hand, a hand-controlled video world model that unifies precise interaction modeling with efficient streaming generation. We curate 100K interaction clips spanning diverse real and synthetic environments and develop hand-aware supervision to improve fine-grained hand motions and object interactions. Building on this foundation, we learn a causal flow map for few-step generation and introduce region-aware score distillation that combines complementary video and image priors to refine interaction dynamics and visual appearance. To further reduce inference latency, we propose Ultra-Fast-VAE, a compact video decoder that preserves high reconstruction fidelity with substantially lower computational cost. Extensive evaluations on HandBench demonstrate superior hand-following accuracy and visual quality compared with existing approaches. Our full-sequence model achieves an FVD of 181.67 and a MANO Dice score of 93.02, while the three-step causal model maintains a MANO Dice score of 92.49. For real-time interaction, 540p streaming on a single H200 reaches 18.03 FPS with 741ms latency and an optimized peak GPU memory footprint of only 34.89GB. We will release the datasets, model weights, and the full training and inference code to support research on hand-controlled video world models.

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Papers" desk, focusing on China's AI industry and domestic foundation model ecosystem. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include LynnReal-Hand, world models, video generation, HandBench.

Source attribution: curated from arXiv cs.AI (@Shaohao Rui, Chaoyang Zhang, Xiaofeng Mao, Weijie Ma).