做多模态模型或视频理解的团队，M³Eval 帮你找到模型记忆的短板，看完你会重新思考模型架构设计。
多模态模型在长视频理解中，记忆能力成为关键瓶颈。现有基准多聚焦感知与推理，缺乏对记忆的系统评估。北京大学团队提出M³Eval，基于认知心理学设计任务，从信息保留、保真度、抗干扰性等维度评估模型记忆。实验发现，模型在并行视频流中难以保持分离表征，时空记忆可靠性差异大，符号记忆有限。该基准为多模态记忆研究提供了重要资源，揭示了模型记忆与人类记忆的显著差异。
M$^3$Eval: Multi-Modal Memory Evaluation through Cognitively-Grounded Video Tasks
As multi-modal models advance towards long-form video understanding, memory emerges as a critical capability. Despite substantial efforts in developing video datasets and benchmarks, existing works primarily focus on perception and reasoning, without systematically evaluating memory: what models retain, how faithfully information is preserved, and how robust memory remains under interference. To address this gap, we introduce M$^3$Eval, the first comprehensive evaluation framework and benchmark for probing different memory dimensions in multi-modal models. Grounded in cognitive psychology, our design features carefully constructed tasks that isolate key aspects of memory. Leveraging M$^3$Eval, we conduct extensive experiments across representative multi-modal models, revealing consistent weaknesses and distinctive behaviors. We find that models struggle to maintain disentangled representations when processing parallel video streams, exhibit interference patterns differing substantially from those observed in human memory, ground memory sources more reliably in the spatial domain than the temporal domain, and demonstrate limited symbolic memory. Collectively, our benchmark provides a valuable resource for future research, while our findings highlight memory as a fundamental yet underexplored capability and offer insights for designing more effective memory mechanisms in multi-modal models. Our code and dataset are available at https://pku-value-lab.github.io/m3eval-homepage.