SWD用不到1%数据就能达到Transcoder的保真，GPT-2、Qwen2.5都能跑。
论文提出稀疏权重分解（SWD），将预训练模型的每个权重矩阵分解为两个稀疏因子，中间坐标作为可寻址电路单元。SWD不需要训练替代网络，在单矩阵替换实验中以不到基线1%的数据量达到与Transcoder等持平的保真度。在GPT-2、Qwen2.5和Qwen3.5-27B上，SWD用更少的读写边和选中单元达到相同电路充分性与必要性目标。SWD还支持微调后全模型替换，并提供零数据变体用于逐步机制可解释性分析。
Sparse Weight Decomposition for Efficient Circuit Extraction
Dense pretrained transformers do not naturally expose interpretable units for circuit extraction. Existing approaches obtain such units by learning auxiliary sparse representations or training sparse models, incurring substantial additional computation while potentially introducing a fidelity gap between the representation being analyzed and the original pretrained model. We propose Sparse Weight Decomposition (SWD), which reparameterizes pretrained linear projections by factorizing each weight matrix into two sparse factors whose shared intermediate coordinates serve as individually addressable circuit units. Without training a separate replacement network, this parametric representation supports the same scoring, selection, and ablation circuit extraction workflow used for methods that learn sparse features. Across single-matrix replacements, SWD matches the held-out fidelity achieved by Transcoder and other strong baselines while using less than 1% of the data that those baselines use to train their replacements. For matched replacement fidelity, SWD reaches the same circuit sufficiency and necessity targets with fewer active read/write edges and selected units across tasks on GPT-2, Qwen2.5, and Qwen3.5-27B. We further show that SWD remains effective for full-model replacement of all attention and MLP weight matrices after fine-tuning the nonzero factor values. Finally, SWD also features a zero-data variant, allowing broader use of mechanistic interpretability analysis (e.g., per-step analysis).