Models

Stanford Marin 开源 Mochi 535B-A23B 全部 57 个预训练中间 checkpoint

Why we picked it

预训练研究者的好素材:Marin 把 535B-A23B 从头训到长上下文阶段的 57 个 checkpoint 全放了,能直接看能力随 token 数怎么涨。想研究 base model 评测、scaling 规律的人可以下载对比,比只拿最终模型有参照价值。

Stanford Marin 团队发布了 MoE 模型 Mochi 535B-A23B 预训练过程中的全部 57 个中间 checkpoint,其中 10 个来自长上下文扩展阶段。这些 checkpoint 基于总计 18.2T tokens 的预训练数据,权重体积约 61 TB,托管在 Hugging Face 的 open-athena 组织下。配套博客发布在 openathena.ai,说明了各阶段的训练细节。

Body

这批中间 checkpoint 的价值在于让研究者可以逐阶段对比模型能力的变化轨迹,解决 base model 评测缺乏参照点的问题。阿里 Qwen 团队的 Justin Lin 转发时也提到,base model 如何评测一直是该领域的难题,这类逐阶段样本对评测研究很有帮助。61 TB 的开源权重对算力有限的团队门槛不低,但适合做 scaling 曲线分析、数据配比消融和长上下文阶段对比等预训练研究。

Original · Junyang Lin

how to eval a base model has been a daunting issue in this field, so if u can share more insights about this that might be even more helpful David Hall @dlwh We’re releasing all 57 intermediate checkpoints from Marin “Mochi” 535B-A23B pretraining, including 10 from long-context extension. 18.2T tokens, about 61 TB of weights. huggingface.co/open-athena/Ma… Blog: openathena.ai/blog/marin-moc… 🔗 View Quoted Tweet📊 5 ⚡

Industry Coordinates & Source Verification

This report is categorized under AITOP's "Models" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include Marin, Mochi, open-source models, pretraining.

Source attribution: curated from Junyang Lin (@JustinLin610).