DeepMem：一种在字节级语言模型上压缩语义并保留字节细节的多尺度架构
Going Deeper With Memories
字节级模型不用分词器但太贵，这篇来自 arXiv 的 DeepMem 用多尺度压缩把计算量砍了约 7600 倍，效果还反超同规模 Transformer，做底层架构的可以看看。
arXiv 论文提出 DeepMem，一个针对字节级语言模型的因果多尺度架构。它用可逆子带变换构建更短的层级序列，通过无损载体和显式细节侧带保留字节信息。在 FineWeb-Edu 和 PG19 上，DeepMem 的语言建模表现超过同等参数量的字节级 Transformer，并在 FRES 语种上取得最优宏平均成绩。瓶颈层的迭代精炼比等效全分辨率推理少约 7,600 倍计算量，同时保持有竞争力的吞吐。它在 PIQA 上提升准确率，代码建模、多项选择推理和字节级损坏鲁棒性也保持竞争水平。
Going Deeper With Memories
Byte level language models avoid tokenizer dependence and provide universal coverage, but processing every byte at full resolution is expensive. We introduce DeepMem, a causal multiscale architecture that separates semantic compression from exact byte preservation. A learned invertible subband transform constructs a hierarchy of shorter sequences, while a lossless carrier and explicit detail sidebands preserve byte identity. Fast weight memory blocks operate along the encoder, with iterative shared state computation at the compressed bottleneck, while an attention based super resolution decoder restores byte resolution representations. DeepMem achieves stronger language modeling performance than a matched byte level Transformer on FineWeb-Edu and PG19, and obtains the best macro average performance across the evaluated FLORES languages. It also improves PIQA accuracy while remaining competitive on code modeling, multiple choice reasoning, and byte level corruption robustness. Bottleneck refinement requires approximately $7{,}600\times$ fewer counted FLOPs than equivalent full resolution reasoning, while maintaining competitive evaluation throughput.