做 NLP 领域适配的团队终于有了比纯 MLM 更优的预训练策略——CLM 绕路法简单有效，在生物医学任务上直接涨点，建议做领域编码器的开发者试试这个两阶段方案。
论文提出一种编码器领域适配新方法：先用因果语言建模（CLM）训练，再切换回掩码语言建模（MLM）微调。在 ModernBERT 上测试，该方法在 8 个法语和 11 个英语生物医学任务上，相比纯 MLM 基线提升 0.3-2.8 个百分点。研究发现 CLM 的密集监督主要影响低层 Transformer 层（0-7），冻结低层会消除收益，而冻结中层则保留收益。这种表征变化在后续 MLM 阶段持续存在，且随模型规模扩大而增强。团队发布了 ModernCamemBERT-bio 和 ModernBERT-bio 作为生物医学编码器新基准。
A Causal Language Modeling Detour Improves Encoder Continued Pretraining
When adapting an encoder to a new domain, the standard approach is to continue training with Masked Language Modeling (MLM). We show that temporarily switching to Causal Language Modeling (CLM) followed by a short MLM decay improves downstream performance. On biomedical texts with ModernBERT, this CLM detour outperforms MLM baselines trained on identical data and compute across 8 French and 11 English biomedical tasks, by +1.2-2.8pp and +0.3-0.8pp respectively, depending on model size. We investigate the reasons for these gains. We find that CLM's dense supervision impacts low transformer layers (0-7) far more than MLM does. Freezing low layers during CLM eliminates the downstream benefit; freezing mid layers preserves it. The representational changes persist through the MLM decay phase, even when it matches the CLM phase in length, and they scale with model capacity. We release ModernCamemBERT-bio and ModernBERT-bio as state-of-the-art biomedical encoders in Base and Large sizes.