MNIST-PRO基准测试揭示了AI智能体在部分可观察环境下的感知能力缺陷，特别是整合碎片化信息和更新错误信念的能力。
MNIST-PRO基准测试将MNIST数字识别转化为基于连续观察的搜索任务，限制了回溯能力。研究人员评估了10个多模态模型在四种记忆表示下的表现，包括原始视觉历史、文本状态、结构化网格地图和整合视觉画布。在部分可观察环境下，模型表现出明显的性能差距，暴露出三个瓶颈：感知状态构建与解释困难、过早停止探索、无法修正早期错误信念。
MNIST-PRO: MNIST is Back as a Partially Observable World for AI Agents
AI agents in partially observable environments need to coordinate active sensing with working memory to maintain an evolving perceptual state. However, existing benchmarks struggle to isolate this perceptual-state construction and interpretation capability because they introduce physical and control complexities. We address this with MNIST-PRO, a benchmark that isolates agentic perception by converting MNIST digit recognition into a sequential, glimpse-based search task with lookback constraints. We evaluate ten multimodal models across four memory representations, including raw visual history, textual states, structured metric grid maps, and a consolidated visual canvas. While models excel under full observability, partial observability exposes a clear performance gap. We identify three distinct bottlenecks. First, perceptual-state construction and interpretation present a challenge, as agents struggle to integrate fragmented glimpses. Second, agents often stop exploring before they see the full sequence. Third, models often fail to revise early, incorrect beliefs even when faced with subsequent contradictory evidence. These results show that simply acquiring visual evidence is not enough. Agents must also be able to build and update a reliable perceptual state.