LiDAR-Hallu基准：4D LiDAR语言模型未必真懂时空关系
Do LiDAR Language Models Really Understand Spatio-temporal Relationships?
做自动驾驶3D感知评测的可以看看，这个基准专抓LiDAR模型靠选项套路蒙对题的毛病，代码和数据都开源了。
研究团队推出LiDAR-Hallu基准，含10,000道题、覆盖150个nuScenes场景，专门检验4D LiDAR语言模型的时空推理。在两个基于B4DL的模型配置上，始终选同一选项就能逼近其多项选择题准确率。对100,000条模型回答的分析显示：仅凭候选答案的时长就能猜对时间类题目，模型在需要相反答案的配对场景中常给出相同回答，且两类配置在所有条件下漏掉全部横向运动正样本。测试的temporal-shuffle对比解码几乎没有净提升，修复的错误被新引入的错误抵消。论文主张评估时空推理应检验模型能否区分被问到的物理关系，而非只看整体准确率。
Do LiDAR Language Models Really Understand Spatio-temporal Relationships?
Recent 4D LiDAR language models aim to reason about objects and their evolving spatial relationships. Yet, in our evaluation, always selecting the same option nearly matches the multiple-choice accuracy of two B4DL-derived configurations. We introduce LiDAR-Hallu, a geometry-referenced benchmark and diagnostic protocol with 10,000 questions across 150 nuScenes scenes. It covers object existence, ego-relative position, distance ordering, relative motion, and temporal localization, with explicit rules for selecting objects, comparing times, and determining reference answers. Our protocol combines fixed-answer and candidate-content controls, cross-scene pairs with identical prompts but opposite reference answers, and relation-specific recall. Analysis of 100,000 recorded responses reveals failures hidden by aggregate accuracy. Candidate duration alone makes temporal answers predictable without observing LiDAR. On paired questions, the models frequently give the same answer to scenes requiring opposite answers. Relation-specific analysis further shows that both configurations miss every positive lateral-motion case across all tested conditions. Temporal-shuffle contrastive decoding provides little net improvement, as repairs are largely offset by new errors and the main failures persist. These results show that evaluating spatio-temporal reasoning requires testing whether models distinguish the queried physical relationships, rather than relying on individual-answer accuracy alone. The source code, checkpoints, and data are released at https://github.com/Awesome4D/4DMLLM_Hallucination_Bench.
