How-toSelected11月6日 04:41

Hugging Face Publishes Comprehensive Guide to Training MoE Models

Leandro's thread: https://t.co/6oAw6ZgUuV

Thomas Wolf@Thom_Wolf

Post

Why we picked it

MoE architectures power many frontier open-weight models, yet training guidance at scale is scarce, so a step-by-step playbook with visual explanations fills a real gap for engineers. Practitioners planning pretraining or fine-tuning runs can use it alongside the team's earlier Ultra Scale Playbook. No regional angle applies; the closest comparison is existing distributed-training documentation from PyTorch and DeepSpeed.

Leandro von Werra of Hugging Face highlighted a new guide by Nouamane on large-scale Mixture of Experts (MoE) training. The guide follows the earlier Ultra Scale Playbook from the same team. According to von Werra, it is the most comprehensive resource available for getting into large-scale MoE training. The material is distributed as an X thread and includes extensive visualisations throughout.

Original · Thomas Wolf

Leandro's thread: https://t.co/6oAw6ZgUuV

Leandro's thread: x.com/lvwerra/status… Leandro von Werra @lvwerra After the Ultra Scale Playbook, Nouamane is back with this crazy guide to train MoEs! It's probably the most comprehensive resource if you want to get into large scale MoE training and an absolute gem packed with beautiful visualization! 🔗 View Quoted Tweet 💬 0 🔄 0 ❤️ 3 👀 675 📊 1 ⚡

#Hugging Face #Mixture of Experts #fine-tuning #open-source models

Keep reading

Recent stories with the same tag or section

How-to BuildersSelected04:42

Hugging Face publishes 30k-word MoE training handbook

The handbook distills hard-won engineering lessons on MoE efficiency that labs usually keep internal, making sparse-model training accessible to smaller teams. Engineers planning distributed pretraining or fine-tuning runs can use its profiling guidance and interactive figures to diagnose bottlenecks without burning hundreds of thousands of GPU-hours themselves. It complements Hugging Face's earlier open training playbooks rather than vendor whitepapers.

#Hugging Face #MoE #fine-tuning #open-source models

Thomas Wolf@Thom_WolfSource

Papers BuildersSelected71°04:41

Hugging Face releases 70,000-word MoE training handbook

Most open MoE knowledge is scattered across engineering blogs and private Slack channels; a consolidated, reproducible guide lowers the barrier for smaller labs to train MoE models instead of falling back to dense architectures. Training teams, ML engineers and researchers can use the runnable code and benchmarked numbers to shortcut months of ablation work. The handbook is hosted free on Hugging Face and sits alongside DeepSeek's own published MoE techniques as a practical reference.

#Hugging Face #Mixture-of-Experts #DeepSeek #fine-tuning

Thomas Wolf@Thom_WolfSource