TTR-Compressor：用测试时回归重做序列压缩
Rethinking Sequence Compression with Test-time Regression
一篇方法论文：把压缩变成回归问题，RULER 成绩超过注意力压缩器，视觉 token 还能砍掉一半多。
arXiv 论文提出 TTR-Compressor，把序列压缩视为测试时回归问题，输入 token 定义回归任务，拟合出的系数即为压缩表示。回归问题通过反向传播端到端学习，系数在前向传播中闭式求解。在块稀疏注意力场景下，TTR-Compressor 在 RULER 基准上大幅超过基于注意力的压缩器；作为多模态大模型的视觉 token 合并器时，用少 55.6% 的视觉 token 达到与标准 MLP merger 相当的效果。
Rethinking Sequence Compression with Test-time Regression
Sequence compression is a fundamental operation and a critical information bottleneck in efficient language and multimodal models, and yet prevailing approaches based on pooling or learned projections remain underperforming. We revisit sequence compression through the lens of test-time regression: input tokens define a regression problem whose fitted coefficients form the compressed representation. Importantly, the regression problem is learned end-to-end via backpropagation, while its coefficients are computed efficiently in closed form during the forward pass. This leads to TTR-Compressor, a general alternative to conventional pooling and projection-based compression. We evaluate TTR-Compressor as the compressor for block-wise sparse attention and the visual token merger for multimodal large language models. On block-wise sparse attention, TTR-Compressor improves RULER performance over attention-based compressors by a large margin. For visual token merging, it consistently outperforms the standard MLP merger and matches its performance with 55.6% fewer visual tokens. These results establish test-time regression as an effective primitive for sequence compression in efficient language and multimodal models.
