MUSE benchmark for vision-language models in education
MUSE: Benchmarking Large Vision-Language Models on Multi-Modal Understanding in Situated Education
This provides a standardized benchmark for evaluating how well AI models understand art and culture in educational contexts, which is crucial for developing more effective AI tools for teaching and learning.
A new benchmark called MUSE evaluates large vision-language models on artistic image understanding in educational settings.
It includes twelve tasks covering visual perception, semantic interpretation, and cultural understanding, with images from Singaporean and Southeast Asian contexts.
Evaluations of open-source and proprietary models show significant disparities, especially in affective interpretation and compositional reasoning.
MUSE: Benchmarking Large Vision-Language Models on Multi-Modal Understanding in Situated Education
Large vision-language models have achieved remarkable progress in multi-modal understanding, yet their capabilities in educational settings remain insufficiently evaluated. In AI-assisted language learning, models must interpret artistic imagery, understand its semantic, affective, and cultural content, and reason about visual context to support meaningful interaction. However, existing benchmarks primarily focus on real-world images or domain-specific educational reasoning, providing limited coverage of artistic educational content. To address this gap, we introduce MUSE, a benchmark for evaluating large vision-language models on artistic image understanding in situated educational applications. MUSE decouples image annotation from question generation, enabling diverse tasks with controllable difficulty while reducing annotation effort. It comprises twelve tasks spanning visual perception, semantic and affective interpretation, culture understanding, and compositional reasoning, together with diverse artistic images deliberately curated to center Singaporean and Southeast Asian multicultural contexts alongside Western art traditions, covering multiple themes and difficulty levels. Evaluation of open-source and proprietary models reveals substantial disparities across capability dimensions, particularly in affective interpretation and compositional reasoning. Our analysis further identifies common failure modes and key challenges for developing trustworthy multi-modal models for education. We hope MUSE will serve as a standardized benchmark for advancing multi-modal understanding in situated educational applications.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include MUSE, vision-language models, benchmark, artificial intelligence.
Source attribution: curated from arXiv cs.AI (@Luyao Zhu, Xun Wei Yee, Wei Li, Mun Thye Mak, Wee Siong Ng).