百度开源 ERNIE 5.5 Flash Base：280B 参数稀疏 MoE 基座模型
百度开源了 280B 参数的 ERNIE 5.5 Flash Base 基座，1M 上下文还带 FP8 权重，想做微调实验的可以下载玩玩。
百度发布 ERNIE 5.5 Flash Base 开源基座模型，总参数 280B，每 token 激活 14B，属于稀疏 MoE 架构。模型预训练使用 33.5T tokens，多语言多领域数据，在同等规模基座模型的多项基准上达到 SOTA。微调和推理支持最高 1M token 上下文，权重提供 FP8 和 BF16 两种格式。权重直接来自预训练，未做 mid-training 或后训练，方便研究者在同一 checkpoint 上对比不同训练方法的效果。
ERNIE 5.5 Flash Base just dropped! Time to open that project folder. 🚀
Our efficient sparse MoE language model draws on data spanning multiple domains and languages, achieving SOTA on multiple benchmarks among similarly sized base models. Its architecture and training system were designed together for efficient training and inference, with advances in attention mechanisms, expert routing, and training objectives: - 280B total parameters; 14B activated per token - Pre-trained on 33.5T tokens - Up to 1M-token context for fine-tuning and inference - FP8 and BF16 weights
The weights come straight from pre-training, with no mid- or post-training. Compare methods from the same checkpoint, measure what each adds, and shape the model around your own tasks.
