这篇论文提出了Persistent SAE，让稀疏自编码器学出特征该持久多久，在解释和监控语言模型上更灵活，尤其长文本场景下效果明显。
标准稀疏自编码器（SAE）独立编码每个token，无法捕捉序列中跨时间的持久信息。论文提出的Persistent SAE为每个特征学习一个持久系数，使模型自动决定哪些特征应持久及持续时长。实验表明，该方法在保持与标准SAE竞争性重建质量的同时，学习到从快速到慢速的特征时间尺度谱：快速特征作为局部可解释检测器，慢速特征在持久状态中集中主题级信息。在提示注入监控案例中，慢速特征能在长上下文中保持检测信号并维持因果有效性。
Persistent Sparse Autoencoders: Learning Feature Timescales in Language Models
Sparse autoencoders (SAEs) decompose language model activations into sparse features, but standard SAEs encode each token independently and do not expose information that persists across a sequence. We introduce Persistent Sparse Autoencoders (Persistent SAEs), which extend standard SAEs by learning a persistence coefficient for each feature, allowing the model to learn which features should persist and for how long. Our experiments show that they retain competitive reconstruction quality while learning a spectrum of feature timescales: fast features behave as locally interpretable detectors, whereas slow features concentrate topic-level information in a persistent state. Moreover, as shown in a prompt-injection monitoring case study, slow features preserve detection signals and remain causally effective over long contexts. These results suggest that Persistent SAEs open up new opportunities for interpreting and monitoring language models through persistent semantic representations.