做多模态情感分析或评估 MLLMs 情感能力的团队，终于有了一个更可靠的多标签基准——MultiEmo-Bench 解决了现有数据集低估模型的问题，值得直接用于模型评测。
本文提出 MultiEmo-Bench，一个多标签视觉情感分析基准数据集，用于全面评估多模态大模型（MLLMs）预测图像引发情感的能力。现有数据集采用单候选情感标注方案，忽略了单张图像可能引发多种不同强度情感的事实，导致低估 MLLMs 能力。新数据集为每张图像雇佣 20 名标注者，收集所有被激发的情感，最终包含 10,344 张图像和 236,998 条有效投票，覆盖八种情感。评估了 Qwen3-VL、GPT、Gemini 和 Claude 等模型，结果表明当前 MLLMs 虽有进步，但仍有很大提升空间。实验还发现，LLM-as-a-judge 方法在视觉情感分析这一主观任务上效果不稳定。
MultiEmo-Bench: Multi-label Visual Emotion Analysis for Multi-modal Large Language Models
This paper introduces a multi-label visual emotion analysis benchmark dataset for comprehensively evaluating the ability of multimodal large language models (MLLMs) to predict the emotions evoked by images. Recent user studies report an unintuitive finding: humans may prefer the predictions of MLLMs over the labels in existing datasets. We argue that this phenomenon stems from the suboptimal annotation scheme used in existing datasets, where each annotator is shown a single candidate emotion for each image and judges whether it is evoked or not. This approach is clearly limited because a single image can evoke multiple emotions with varying intensities. As a result, evaluations based on these datasets may underestimate the capabilities of MLLMs, yet an appropriate benchmark for evaluating such models remains lacking. To address this issue, we introduce a new multi-label benchmark dataset for visual emotion analysis toward MLLMs evaluation. We hire $20$ annotators per image and ask them to select all emotions they feel from an image. Then, we aggregate the votes across all annotators, providing a more reliable and representative dataset labeled with a distribution of emotions. The resulting dataset contains $10,344$ images with $236,998$ valid votes across eight emotions. Based on this benchmark dataset, we evaluate several recent models, including Qwen3-VL, OpenAI's GPT, Gemini, and Claude. We assess model performance on both dominant emotion prediction and emotion distribution prediction. Our results demonstrate the progress achieved by recent MLLMs while also indicating that substantial room for improvement remains. Furthermore, our experiments with LLM-as-a-judge show that the method does not consistently improve MLLMs' performance, indicating its limitations for the subjective task of visual emotion analysis.