做多模态模型评测的团队终于有了对抗感知偏见的方法——Perceptual Perturbation 框架能直接提升评估者的视觉可靠性，建议做 MLLM 评测基准的开发者点开看看实验细节。
研究发现多模态大语言模型作为自动评估者时存在“感知判断偏见”：当视觉证据与文本线索冲突时，模型倾向于奖励看似合理但感知错误的回答。研究者通过受控视觉扰动构建了 Perceptually Perturbed Judgment 数据集，并开发了结合 GRPO 奖励与批量排序目标的统一训练框架。该方法无需显式成对标签，即可实现全局一致性排序。实验表明，该方法显著提升了多模态评估者的感知保真度、排序一致性与人类评价对齐度。这项工作为训练感知可靠、可解释且鲁棒的多模态评估者提供了可扩展路径。
Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling
Recent multimodal large language models have demonstrated strong reasoning ability, yet their reliability as automated evaluators remains limited by a critical weakness: when visual evidence conflicts with textual cues, MLLM judges tend to reward plausible narratives over perceptually correct answers. We identify and systematically analyze this phenomenon, which we term Perceptual Judgment Bias. Through controlled visual perturbations, existing multimodal judges frequently anchor on the response text instead of their own visual perception, leading to inconsistent and non-verifiable evaluations. To address this issue, we introduce the Perceptually Perturbed Judgment Dataset, which constructs minimally edited counterfactual responses that isolate perceptual errors and enable verifiable supervision. Building on this dataset, we develop a unified training framework that combines a structured GRPO-based reward with a batch-ranking objective, achieving coherent global ordering without explicit pairwise labels. Experiments across diverse MLLM-as-a-Judge benchmarks show that our approach substantially improves perceptual fidelity, ranking coherence, and alignment with human evaluation. Our results establish a scalable and generalizable pathway for training multimodal judges that are perceptually grounded, interpretable, and robust to visual-reasoning conflicts.