这篇论文搞了个空间概念测试，专门考大模型的方向、距离和组合推理，结果发现它们比想象中笨。
该论文提出 Geo-Spatial Concept Probing 基准，针对方向、距离、拓扑等空间概念及其组合进行受控测试。他们使用问答任务作为代理，在多种 LLM 架构和训练设置下开展实验，分析模型规模与设计的影响。结果显示当前 LLM 在概念的抽象性、组合性和基础性上存在明显局限，代码将在 GitHub 公开。
Geo-Spatial Concept Probing of Large Language Models: Abstraction, Compositionality, and Grounding
Understanding concepts is fundamental to generalization. Despite their impressive performance on a wide range of tasks, Large Language Models (LLMs) still struggle with genuine concept understanding. Prior work has evaluated conceptual understanding in LLMs using natural-language benchmarks or narrowly scoped synthetic tasks, but these settings often conflate multiple skills or lack precise control over the underlying concepts and their properties. To support controlled probing of concepts in LLMs, we design tests on their core properties: abstraction, compositionality, and groundness. We set up a concept-centric benchmark, targeting spatial concepts such as direction, distance, topology, and their compositions, and use question answering tasks serving as a proxy. We conduct extensive experiments across multiple LLM architectures and training regimes to analyze how model scale and design impact conceptual understanding. The results reveal clear limitations in current LLMs and provide insights into the factors shaping their ability to acquire and compose structured concepts. Our findings shed light on how concept-based LLMs can be redesigned for improved information access and knowledge management. The code will be available at https://github.com/rd20karim/concept-probing.