这篇论文用大语言模型写代码来初始化强化学习，不用模拟器和预收集数据，在Gym和Control Suite上都能省样本。
ProDVI是一个利用大语言模型为强化学习智能体提供初始化先验的新框架。它让代码生成模型输出Python函数，编码对环境动力学的粗略假设，再生成合成转移数据。这些数据用于预训练actor-critic价值网络的状态-动作编码器，使智能体在在线强化学习开始前获得动力学感知的表示。在OpenAI Gym和DeepMind Control Suite任务上，ProDVI显著提升了无模型强化学习算法的样本效率。
ProDVI: Programmatic Dynamics Priors for Value Network Initialization
Deep Reinforcement Learning (RL) is notoriously sample inefficient. One contributing factor is that RL agents are typically initialized from scratch, forcing them to acquire task-relevant knowledge through online interaction. Existing approaches obtain informative initializations through pre-collected datasets, high-fidelity simulators, or meta-learning over related tasks, but these prerequisites may be difficult to access or even unavailable. In this paper, we propose Programmatic Dynamics Priors for Value Network Initialization (ProDVI), a framework that leverages the commonsense and domain knowledge encoded in large language models to initialize RL agents without relying on these resources. Specifically, ProDVI prompts a code-generating language model to produce executable Python functions that encode coarse hypotheses about environment dynamics. These functions are then used to generate synthetic transitions. Based on these transitions, we construct an auxiliary dynamics prediction objective to pretrain the state-action encoder of the value network in an actor-critic framework. The learned representation provides dynamics-aware inductive biases before online RL begins. Importantly, the generated programs are used only for representation pretraining and are not required to faithfully simulate the target environment. While the generated programs may be inaccurate, their induced initialization can be corrected through online learning from real transitions and rewards. Experiments on OpenAI Gym and DeepMind Control Suite tasks show that ProDVI can effectively improve the sample efficiency of model-free RL algorithms.