World Modeling in Transformers
This paper provides a detailed mechanistic analysis of how a transformer model represents and navigates a simulated environment, which could be useful for researchers working on similar tasks.
A new paper analyzes TaxiGPT, a transformer trained on Manhattan maps, showing it represents intersections and streets but fails due to interference between superposed features.
Mechanistic analysis reveals the model tracks position and uses a goal compass for navigation.The study proposes indicators to compare models and suggests shifting focus from whether a model has a world model to studying its capacities.
Behavioral failures can make a transformer appear to lack a world model even when it has learned faithful representations of its environment. We demonstrate this in TaxiGPT, a transformer trained on random walks through Manhattan whose failures have been interpreted as evidence of an incoherent internal map. Through mechanistic analysis and causal interventions, we show that the model represents intersections and streets, tracks its position, and uses a goal compass to navigate. We trace its failures to interference between superposed intersection features, which disrupts localization within the internal map. Affordance packing, which groups representations of intersections with the same legal moves, helps limit the consequences of these errors. Finally, we propose mechanistic indicators that we use to compare models and show that world-modeling capacities emerge at different stages of training. Our findings motivate a shift from asking whether a model has a world model to mechanistically studying its world modeling: the interacting capacities through which it represents its environment and uses those representations to guide behavior.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and AI regulation, compliance frameworks, and sovereign compute policy. Key entities and technical topics under continuous tracking include Transformer, TaxiGPT, world models.
Source attribution: curated from arXiv cs.AI (@Pierre Beckmann, Matthieu Queloz, Andre Freitas).