De-biased VLM-as-3D-Judge协议用于单图像3D生成优化

Judging to Improve: A De-biased VLM-as-3D-Judge Protocol for Single-Image 3D Generation

精选理由

这篇论文介绍了一种让AI自动评判并优化3D生成质量的方法,不用人工打标签。它把VLM评委从打分变成训练的一部分,在特定类别(家具)上达到了和强基线持平的效果,但说明了简单微调还不够。

AI 摘要

该论文提出了一种去偏见的、跨模型的VLM-as-3D-Judge协议,用于可靠地评估单图像到3D网格质量,弥补了传统几何和CLIP代理的不足。研究者将该评分器用于优化开源生成模型TRELLIS在特定资产类别(家具)上的表现,无需人工标注。他们使用训练评委(Qwen2.5-VL-7B)和评估评委(InternVL3-8B)分离以避免循环,并修复了图像过载、几何隐藏渲染和无参考评分等三种失败模式。实验表明,在轻度参数高效适应下,该方法与强基线持平(胜率0.50),但无法超过65%的胜率目标,且发现干净输入会饱和评委。

原文 · arXiv cs.LG

Judging to Improve: A De-biased VLM-as-3D-Judge Protocol for Single-Image 3D Generation

A companion study established a de-biased, cross-model VLM-as-3D-judge that reliably ranks single-image-to-3D mesh quality where cheap geometry and CLIP proxies fall short. This paper asks: can that judge's preferences specialize a strong open generator, TRELLIS, on one asset class (furniture), cheaply and without human labels? Taking the judge from ranking to optimization is where the work lives. Pushing a VLM judge into the training and evaluation loop exposes failure modes ranking never triggered, so our contribution is an optimization-grade hardening of the judge: a training judge (Qwen2.5-VL-7B) held distinct from an evaluation judge (InternVL3-8B) to break circularity; position-bias correction; and fixes for three failure modes (image overload, geometry-hiding splat renders, and reference-free judging that rewards clean-but-wrong outputs), with calibration evidence (clear-gap win-rate 0.83-1.0; base-vs-base ~0.5). Using this protocol as an independent evaluator, and working only from public models and data with lightweight parameter-efficient adaptation, we find our methods match the strong base rather than exceed it. Independent base samples carry essentially no learnable preference (0.94 order-flip rate), so signal must be engineered by quality-contrastive construction. Across six adaptation methods, two input regimes, and a severity sweep, the most targeted - conditioner repair under severe degradation - reaches parity (0.50) with the base, while no method clears the >=65% win-rate target. The result is mechanistic: clean inputs saturate the judge, flow-DIT fine-tuning washes out through the sampler, and conditioning repair is the locus that moves geometry. Win-rates are directional at n=8 objects. Matching a strong public-data base with cheap adaptation is itself informative: exceeding it needs more than lightweight PEFT on public data, and the judge protocol is reusable.