这篇论文给出了一个能精确测量Mamba状态使用情况的工具,还发现模型会根据输入动态调整哪些状态起作用——基于这个发现做剪枝,效果比之前的方法好很多,7B模型上预算减半性能不变。
研究团队为Mamba等选择性状态空间模型开发了一种精确测量状态使用的工具,该工具基于对角状态矩阵将输出分解为每个模态贡献,并通过Gram张量计算任意子集剪枝的精确误差。在Mamba-1家族(130M至2.8B参数)上验证,与参考实现相对误差仅2.3×10⁻⁷;该工具预测一层剪枝误差的中位相对偏差为5×10⁻⁷(基于4,464个配置)。分析Mamba-1、已部署的7B Falcon-Mamba及Mamba-2发现,模型会随输入重新分配状态空间,哪些模态承载信号会随上下文迁移;基于输入调度的模态剪枝在各级别(130M至7B)均优于静态、Hankel及自适应排名,在半预算下匹配未剪枝模型性能。
An Exact Instrument for State Usage in Selective State-Space Models, and the Input-Driven Migration It Reveals
Selective state-space models such as Mamba route information through a bank of first-order modes whose input coupling is set by a learned selection mechanism. We give an exact instrument for measuring how a trained model uses these modes. Because the state matrix is diagonal, each channel's output decomposes exactly into per-mode contributions, and a per-(layer, channel, window) Gram tensor yields the exact output error of dropping any subset of modes, offline, at any budget. Validated against the reference implementation to a relative error of $2.3\times10^{-7}$ on the Mamba-1 family where it is exact, the instrument predicts a layer's deployed pruning error to a median relative deviation of $5\times10^{-7}$ over $4{,}464$ configurations, its floor set by the reconstruction. Applying the instrument across the Mamba-1 family (130M--2.8B), the deployed 7B Falcon-Mamba, and Mamba-2, we find that trained models re-allocate their state space with the input: which modes carry the signal migrates across contexts, and at the most affected layers a per-input oracle roughly halves the output error of a fixed mode set. Frozen-signal counterfactuals attribute the migration primarily to the input-dependent write map $B_t$; the timestep usually identified with selectivity carries almost none of it. Input-scheduled mode pruning on this measurement outperforms static, Hankel-based, and layer-adaptive rankings at every scale from 130M to the deployed 7B Falcon-Mamba, and at half the state budget it matches the unpruned model. Because the scheduler reads each window's mode usage from a first pass, this demonstrates realizable headroom; we claim no deployed compute or memory saving.