做 LLM 微调的团队终于有了评估预训练能力保持的基准——PEFT-Arena 帮你选方法时不再只看下游指标，建议做模型适配的开发者点开看看。
论文提出 PEFT-Arena 基准，从稳定性-可塑性困境（目标任务适应 vs 预训练能力保持）评估参数高效微调方法。研究发现，在同等参数预算下，正交微调在帕累托前沿上表现最佳。通过权重空间谱分析和激活空间表示保持分析，解释了不同方法遗忘预训练能力的原因。最终检查点往往偏离最优平衡点，论文展示了路径回退的后处理改进案例。
PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective
Parameter-efficient finetuning (PEFT) has become the standard approach for adapting large language models, yet evaluations largely emphasize downstream accuracy while overlooking the retention of pretrained capabilities. We argue that PEFT should be assessed through the stability-plasticity dilemma: the trade-off between target-task adaptation and resistance to forgetting. We introduce PEFT-Arena, a benchmark that jointly measures downstream performance and general capability retention. Across methods, we find distinct stability-plasticity profiles; under comparable parameter budgets, orthogonal finetuning achieves the most favorable Pareto frontier. To explain these differences, we analyze PEFT updates from two geometric perspectives. In weight space, spectral analysis reveals how parameterizations interact with the pretrained singular-value structure. In activation space, retention metrics show whether finetuning preserves or distorts general-capability representations, with forgetting linked to non-isometric representation distortion. Finally, an analysis shows that final SFT checkpoints often overshoot a better target-retention operating point. Inspired by this, we present case studies of a post-hoc improvement with path-wise rewinding.