这篇论文拿彩色方块网格考OpenAI的多模态模型，让它们看图写话再画图，结果画不对，做自动GIS得先解决这问题。
论文提出一种模态转移任务，要求LMM先用文本描述规则网格中的彩色方块图像，再用新的LMM实例根据文本描述重新生成原空间场景。实验采用OpenAI的近期LMM，结果显示其在重新生成简单彩色方块网格图像时仍存在困难。研究表明，LMM的强健地理空间理解需要严格的多模态对齐，当前模型在图像和文本模态间转移空间信息的能力仍是瓶颈。
LMM Modality Transfer: A Pre-requisite for Autonomous GIS Agents
AI models are becoming increasingly adept at understanding and processing spatial information, thereby facilitating agentic problem-solving in spatial tasks and workflows. However, most of the research on their spatial capabilities (e.g., spatial reasoning) has focused on the textual modality as input and output. This contrasts with the human approach to GIS workflows, where text and visual modalities are often used together, interchangeably, and in a complementary manner. Thus, to truly achieve an automated GIS analysis pipeline or carry out human-designed GIS workflows, AI models --- Large Multimodal Models (LMMs) in particular --- need to be able to seamlessly transition between image- and text-based modalities that are traditionally used in such workflows. We present a modality transfer task that (1) asks an LMM to first describe an input image of colored squares in a regular grid, and (2) asks a new LMM instance to re-generate an image of the original spatial scene using the textual description output by the former model. This task quantifies the ability of LMMs to transfer spatial information between image and text modalities. Ultimately, by examining the modality transfer capability of LMMs through the lens of spatial information theory, this work highlights a critical bottleneck: achieving strong and robust geospatial understanding in LMMs requires rigorous, multi-modal alignment. Our results indicate that recent LMMs (here from OpenAI) still struggle with modality transfer, when tasked with re-generating an image of a simple spatial grid of color squares.