Video Generative Models as Geometry Learner
GeoNeXt achieves state-of-the-art results in depth and surface normal estimation with minimal training data, offering a more efficient approach for researchers and developers working with limited datasets.
The paper introduces GeoNeXt, a method that repurposes pretrained video generative models for geometry estimation.
It treats geometry estimation as a next-frames prediction task, enabling joint modeling of images and geometry targets.
GeoNeXt outperforms previous task-specific and unified generative competitors while using substantially less training data.
The method rivals discriminative state-of-the-art approaches trained on over 100x more data and excels on several benchmarks.
Recent generative approaches to geometry estimation adapt pretrained image diffusion models and treat the task as image-conditioned generation. Leveraging off-the-shelf image diffusion models, they either (i) train task-specific geometry models (for depth and surface normal estimation) independently, losing the opportunity of exploring the intrinsic correlation of these geometric targets, or (ii) jointly fine-tune modified image diffusion backbones (e.g., altered self-attention), which typically demands substantial labeled data. To overcome these limitations in a principled fashion, we repurpose pretrained video generative models as a unified and data-efficient framework for geometry estimation, formulated innovatively as a next-frames prediction task. Our method, GeoNeXt, inherits naturally structured knowledge and richer priors from the video model, while further adapting them for joint modeling of images and geometry targets (image <-> geometry), enabling more data efficient and effective learning of geometry. Extensive experiments validate our method for zero-shot monocular depth and surface normal estimation across diverse datasets, outperforming both previous task-specific and unified generative competitors while using substantially less training data. Notably, our method rivals discriminative state-of-the-art approaches trained on over 100x more data and even standouts on several benchmarks.
This report is categorized under AITOP's "Papers" desk. Key entities and technical topics under continuous tracking include GeoNeXt, video generation models, depth estimation.
Source attribution: curated from arXiv cs.AI (@Haosen Yang, Jifei Song, Zhensong Zhang, Xiatian Zhu, Jiankang Deng).
