Sparse MoE Routing Geometry Study
The findings help optimize MoE model efficiency by revealing reusable routing processes across layers, potentially reducing computational costs.
The findings help optimize MoE model efficiency by revealing reusable routing processes across layers, potentially reducing computational costs.
这篇论文用OLMoE和DeepSeek-V2-Lite实测证明,MoE路由的选专家和加权输出不该共用一套分数,FDAA方法能白捡性能提升,做MoE优化的值得看。
这篇论文把SMoE的不连续性分析得清清楚楚,还附带了一个计算量很小的平滑方法,实验证明又能提高效果,值得一看。