研究比较 Transformer 与 Mamba 的证据位置偏差
Mixing Matters: Evidence Position Bias Across Sequence Mixers in Long-Context Question Answering
把同一份证据在不同位置挪来挪去，测出 Transformer 有开头优势而 Mamba 没有，做长文本应用前值得看看这些曲线。
该论文在长文本问答任务中将一份含答案的文档在 9 份干扰文档中移动，比较 Transformer、状态空间模型和 hybrid Mamba 架构的位置偏差。被测 Transformer 表现出开头优势，状态空间模型则没有，但两类模型都偏爱靠近末尾的证据。Mamba 纯架构与混合架构的对比方向一致但统计上不确定。研究还发现位置信息在各架构中都可解码，注意力在初始 token 附近的集中与 Transformer 的开头敏感度相关。
Mixing Matters: Evidence Position Bias Across Sequence Mixers in Long-Context Question Answering
Long-context models often use answer-bearing evidence less effectively when it appears away from a prompt boundary. We compare this position bias across Transformer, state-space, and hybrid Mamba architectures by moving one gold document through a fixed set of nine distractors. The evaluated Transformers show a beginning advantage that the evaluated state-space models lack, while both families favor evidence near the end. A matched pure-versus-hybrid Mamba comparison points in the same direction but remains statistically uncertain. The family difference appears among models that can answer the task. Transformer primacy reappears on synthetic retrieval, where state-space ceiling prevents a family comparison. A corpus swap within one state-space architecture changes overall accuracy without a detectable change in curve shape. Prompt changes can create or nearly remove a beginning advantage. Position remains decodable across architectures even when response curves differ, and attention concentration near initial tokens covaries with beginning sensitivity in the tested Transformer family. These exploratory results show that effective context use depends on the evaluated architecture family, capability, task, and prompt, without isolating the sequence mixer as the sole cause.