这篇论文提出用LoRA和混合专家架构让智能体不断学新动作而不忘旧的,5任务基准上几乎零遗忘,值得做持续学习或运动生成的看看。
本研究提出基于冻结大语言模型和LoRA变体的持续学习方法,使运动-语言智能体能在不遗忘旧知识的情况下增量学习新动作概念。采用混合专家架构和基于自编码器的路由器在推理时选择任务特定专家,无需任务标签。在从HumanML3D导出的5任务基准上,该方法在运动到文本(M2T)和文本到运动(T2M)两个方向上实现了近乎零遗忘,且保持高质量生成与描述。实验表明硬专家选择在质量指标上显著优于软专家混合,专家隔离对持续学习性能至关重要。
Towards Continual Motion-Language Agents: LoRA Variants for Incremental Motion Understanding and Generation
Motion-language agents must possess the bidirectional capability to both understand human movement (motion-to-text, M2T) and generate it from natural language (text-to-motion, T2M). While foundational models have achieved strong performance in static settings, autonomous agents operating in dynamic environments must continuously incorporate new motion concepts -- such as novel athletic styles or specialized gestures -- without catastrophic forgetting of previously acquired skills. We investigate the stability-plasticity trade-off in bidirectional motion-language learning under sequential task exposure. Building on a frozen large language model backbone, we introduce low-rank adaptation (LoRA) variants designed to mitigate inter-task interference. We specifically propose mixture-of-experts architectures that utilize an autoencoder-based router to select task-specific experts at inference time, so that no task-label is needed. To evaluate these methods, we establish a reproducible five-task benchmark derived from HumanML3D through semantic clustering of motion descriptions. Our experimental results demonstrate near-zero forgetting across both M2T and T2M directions while maintaining high generation and captioning quality. Furthermore, we show that hard expert selection via routing significantly outperforms soft expert blending in quality metrics, indicating that preserving expert isolation is critical for maintaining performance in our continual learning setting. Finally, we observe that a divergence between token-level accuracy and downstream generation quality may occur, highlighting the need for more comprehensive evaluation protocols in future research on lifelong motion-language agents.