Anthropic Superposition Model Phases
Emergent phases of superposition: from partial to full representation
This research reveals how model architecture and data statistics jointly determine representation efficiency in large models. The findings help explain why wider models can represent more features without proportional loss increases, impacting how we scale AI systems. For developers and researchers working with large language models, these insights can guide architecture decisions and data preprocessing strategies to optimize representation capacity.
Anthropic's toy model demonstrates a phase transition in superposition as model width increases.
The transition occurs from partial-representation to full-representation phases.Critical width grows linearly with active features up to a logarithmic factor.
Loss scaling changes across phases: linear growth below critical width, quadratic growth above it.Non-uniform firing probabilities delay transition and reduce loss.
Emergent phases of superposition: from partial to full representation
Large language models are thought to represent features by vectors in a hidden space of dimension given by the model's width. Superposition, in which more features are represented than the width by letting representation vectors overlap, is a leading account of how representation vectors are organized. However, how model width and data statistics determine the configuration of representation vectors and the resulting loss when the number of features and the width are large remains less understood. Here we show, in Anthropic's toy model of superposition, that increasing the width drives a continuous phase transition from a partial-representation phase, where only a subset of features receives appreciable representation vectors while the rest vanish, to a full-representation phase, where every feature is represented. Our theory via a partial random projection approximation predicts, and experiments confirm, that the critical width grows linearly with the number of active features up to a logarithmic factor. The loss scaling changes across the transition: below the critical width, the loss grows linearly with the number of active features and depends weakly on the width in a form set by data statistics; above it, the loss grows approximately quadratically with the number of active features and decays inversely with the width. Non-uniform firing probabilities delay the transition and lower the loss, as more frequent features occupy more space. Our results provide an account of how model width and data statistics jointly shape representations and loss, a step toward understanding representation scaling in large models.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include Anthropic, superposition, representation scaling, phase change.
Primary source verification: published directly by arXiv: Anthropic with first-party authority. Cross-source corroboration: 4 independent sources have covered this development (including The Rundown AI, AI Will, Simon Willison’s Weblog), confirming industry-wide signal strength.
