MoE 路由干预研究:专家路径如何影响 Qwen3-30B 输出
What MoE Routing Controls: Causal Interventions on Expert Trajectories
这是篇挺有意思的可解释性工作:直接给 Qwen3-30B 的路由加偏置就能控制输出格式。搞 MoE 或可控生成的人可以看看,中间层才是关键这个发现挺有用。
arXiv 论文(编号 2611.22120)研究了混合专家模型中的路由控制问题,实验对象为 Qwen3-30B-A3B。研究方法是在生成时向目标专家的 router logit 加入固定偏置,把 token 强制引导到由某概念或角色常用专家构成的目标路径上。
研究发现两类目标的行为差异明显:格式路径(format path)被引导后能实际改变模型写出的文本,而概念路径(concept path)虽能把 next-token 概率拉向该概念——即使路径是从另一语言构建的——但对最终生成文本影响很小。作者通过分段生成实验找到解释:格式一旦开始就会自我强化。实验还显示 religion 和 money 两个概念引导效果最强,且效应集中在模型中间层,作者进一步用嵌入区分度和词频来解释不同概念间效应强度的差异。
What MoE Routing Controls: Causal Interventions on Expert Trajectories
In mixture-of-experts (MoE) language models, a token is routed to a sequence of experts across layers. We investigate how forcing tokens onto chosen expert paths influences model behavior. We build a target path from the experts that a concept's or role's tokens most often use, and we add a fixed bias to each target expert's router logit during generation to steer the token toward this expert path. We study two kinds of targets. Format targets concern the form of the output, and concept targets concern its topic. In Qwen3-30B-A3B, we show that steering a format path changes the text the model writes. We also show that steering a concept path shifts the model's next-token probabilities toward the concept, even when the path is built in another language, yet this shift has little effect on the generated text. We investigate why formats reach the text but concepts do not by steering only part of the generation, and find evidence that a format, once started, reinforces itself. We show that the concepts religion and money steer most strongly, and that the effect is concentrated in the middle layers. We attempt to explain why the strength of the effect varies by concept through an analysis of embedding distinctiveness and word frequency.
This report is categorized under AITOP's "Papers" desk, focusing on China's AI industry and domestic foundation model ecosystem. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include MoE, Qwen3, interpretability.
Source attribution: curated from arXiv cs.LG (@Mukund Athmaram, Manoj Rao, Liang Ma).