PlayGen2 Real-Time World Model Uses Explicit 3D State
PlayGen2: Learning Real-Time Playable Worlds with Explicit 3D State
The architectural split between 3D state prediction and rendering avoids the drift that plagues video-frame-recurrent world models, and the laptop-scale latency shows the approach runs on consumer hardware rather than data-centre GPUs. Developers building playable-world or interactive-simulation demos can download the code and weights and extend the framework to other games or rule-based gameplay. It compares with video-only world models like Genie-style systems by trading appearance realism for spatial persistence and control.
PlayGen2 is a real-time interactive world model that separates learned 3D state evolution from neural rendering, released with code and pretrained weights. It maintains a persistent world memory with a volumetric occupancy grid and identity-indexed 3D states for players and entities. Tested on Doom, it learns geometry-aware player motion and player-NPC interactions, achieving roughly 160ms backend latency on a MacBook Pro with an M4 Pro chip. It also supports zero-shot play on unseen Doom maps, multiplayer interaction by composing single-player models, and programmable gameplay via rule-based state interventions.
PlayGen2: Learning Real-Time Playable Worlds with Explicit 3D State
Building a playable world requires more than generating plausible video: it demands responsive control, persistent spatial structure, and coherent interactions with dynamic entities. Using generated frames as the sole recurrent world representation ties these requirements to view-dependent appearance synthesis, making it difficult to maintain long-term spatial and temporal consistency in generation and interaction. We present PlayGen2, a real-time interactive world model that separates learned 3D state evolution from neural rendering. PlayGen2 maintains a persistent world memory comprising static geometry represented by a volumetric occupancy grid and identity-indexed 3D states for the player and other entities. Within a fixed-size local region, its world-state model jointly learns action-conditioned entity dynamics, geometry completion, and entity discovery, committing updates to a persistent world memory. A separate neural renderer synthesizes first-person observations from depth and semantic projections of the world state, without feeding generated images back into state prediction. Using Doom as a testbed, we demonstrate that the model successfully learns world dynamics including geometry-aware player motion and player--NPC interactions. The framework is computationally efficient, enabling the complete simulation and rendering pipeline to achieve approximately 160ms backend latency on a MacBook Pro with an M4 Pro chip. Beyond single-player interaction, we demonstrate zero-shot deployment on unseen Doom maps, multiplayer interaction through the composition of single-player models, and programmable gameplay through rule-based state interventions. These capabilities demonstrate the transferability and extensibility of the learned game-world model. Code and pretrained model weights are publicly available.
