Stanford Marin 团队发布 Marin Mochi 535B-A23B 检查点及技术报告
Stanford Marin 团队放了个 535B 总参、23B 激活的 MoE 模型检查点,技术报告把每个架构组件为什么这么选都讲透了。做模型训练和架构研究的人值得翻报告,XSA 这类新设计能看到第一手分析。
Stanford Marin 团队放出 Marin Mochi 的 535B-A23B 检查点,即总参数 535B、每 token 激活约 23B 的稀疏 MoE 模型,同时附上详细技术报告。报告解释了各组件的选择依据,主要围绕训练稳定性和 GPU 吞吐量两个目标展开。
架构上有几处设计值得注意:采用 latent MoE,并在 latent 层额外加了一个 RMSNorm,使 latent MoE 方案真正跑通;注意力部分使用经典的 3:1 滑动窗口/全局 gated GQA 组合,局部层用 partial RoPE、全局层用 NoPE;同时在 keys、attention 输出和 MoE 输出上加了短卷积。此外还引入了 XSA(Exclusive self-attention), Sebastian Raschka 评价这是今年年底架构盘点不会无聊的原因之一。
这类把每个组件的取舍理由写清楚的技术报告,对做模型架构研究和训练工程的团队参考价值较高,比单纯放 checkpoint 更有信息量。
Whoa, great and even excessive Marin Mochi (535B-A23B) checkpoint drop (in a good way!)
Comes with great tech report btw with insights into why / how components were picked (for training stability and GPU throughput).
Short fact sheet:
1) Latent MoE with an extra RMSNorm on the latent that made latent MoE actually work (better)
2) Classic 3:1 sliding-window/global gated GQA, partial RoPE (local) and NoPE (global)
3) Adding short convolutions on keys, attention output, and MoE output
4) Exclusive self-attention (XSA) -- I guess the end-of-the year architecture roundup this year won't be boring 😅
This report is categorized under AITOP's "Models" desk, focusing on US frontier AI labs and Silicon Valley technology leaders. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows and US-China foundation model benchmarks and commercialization comparison. Key entities and technical topics under continuous tracking include Marin Mochi, Marin, MoE, open-source models.
Source attribution: curated from Sebastian Raschka (@rasbt).