AgentSTAR：用 VLM 智能体从单目视频做形状重建与跟踪
AgentSTAR: Agentic Shape Tracking and Reconstruction from Monocular Videos
让 VLM 智能体边渲染边比对来跟踪 3D 物体，ARCTIC 和 HOT3D 两个基准上都跑赢了现有方法。
论文提出 AgentSTAR，一种从单目视频进行物体形状重建与跟踪的方法，采用智能体式的分析合成流程。方法先推断结构化 3D 物体模型，包括几何和运动学结构，再由 VLM 智能体在渲染-比较循环中迭代优化形状或广义位姿，把粗粒度视觉推理和数值位姿优化结合起来。不同于先估计密集像素对应再恢复运动的传统路线，AgentSTAR 不依赖像素匹配目标，能在大幅运动、关节活动和严重遮挡下持续跟踪。在 ARCTIC 基准上，它大幅超过关节物体 3D 点跟踪的最先进基线；在 HOT3D 上超过所有被评估的刚体跟踪基线。
AgentSTAR: Agentic Shape Tracking and Reconstruction from Monocular Videos
In this work, we present a method for shape reconstruction and tracking from video via agentic analysis-by-synthesis. Unlike prior methods which first estimate dense pixel correspondences and then recover object motion from them, our method infers a structured 3D object model, including its geometry and kinematic structure, and uses this model to optimise object track estimates over time. In our optimisation loop, a Vision-Language Model (VLM) agent iteratively refines shape or generalised pose through a render-and-compare loop, combining coarse visual reasoning with numerical pose optimisation for precise state estimation. This structured formulation enables our method to track through large motion, articulation, and severe occlusion without relying on pixel-matching objectives. Quantitatively, on ARCTIC, our method substantially outperforms state-of-the-art 3D point-tracking baselines for articulated objects, and on HOT3D it outperforms all evaluated rigid-object tracking baselines.
