GEM-MPC: Balancing Exploration and Exploitation through Expert-Guided Planning
For researchers working on reinforcement learning, this paper presents a new method that improves the balance between exploration and exploitation in high-dimensional control tasks.
This paper introduces GEM-MPC, a reinforcement learning method that improves planning-based methods by combining a policy trained to clone the planner with a KL-regularized policy for exploration.
It uses MPPI and introduces Gated Prior Distillation to reduce computational cost.Across continuous-control benchmarks, GEM-MPC outperforms existing planning-based baselines.
Effective exploration in high-dimensional continuous control remains a central challenge in reinforcement learning. Planning-based methods address this by combining online planning with learned policies and value functions, but their components can become misaligned during training: learned sampling policies may diverge from planner behavior, while planning distributions stored in replay become stale as the model and value function evolve. Reanalysis can refresh these targets, but at substantial computational cost. We propose GEM-MPC, an MPPI-based reinforcement learning method that improves the interaction between planning and learning. GEM-MPC uses MPPI to combine a policy trained to clone the planner with a KL-regularized policy that explores around it, providing complementary exploitation and guided exploration within planning. We further introduce Gated Prior Distillation, which selectively learns from stored planning distributions only when they provide a better target than the current prior, reducing the impact of stale planning data without requiring full reanalysis. Across continuous-control benchmarks, GEM-MPC consistently outperforms existing planning-based baselines under lower computational budgets.
This report is categorized under AITOP's "Papers" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include reinforcement learning, planning-based methods, GEM-MPC, continuous control.
Source attribution: curated from arXiv cs.LG (@Alvaro Serra-Gomez, Thomas Moerland).