音乐Transformer换个调就认不出旋律，EMT用自蒸馏让它真正学到结构，生成效果还比SOTA好。
人类能识别时移或转调的音乐，但标准音乐Transformer对这类输入会产生不相关表示。研究发现模型规模增大或训练时间更久时，等变性反而下降。为此提出EMT，通过自蒸馏联合优化下一token预测与辅助等变正则化损失。等变损失作为正则化器，同时提升预测能力和表示质量。在客观与主观评估中，EMT均超过数据增强、特征工程和SOTA基线。
Equivariant Music Transformer
Humans recognize a musical passage even when it is shifted in time or transposed in pitch, indicating a notion of equivariance in the representation space. Our analysis, however, shows that standard music transformers map such time-shifted or pitch-transposed inputs onto uncorrelated representations: these models become progressively less equivariant as they scale in size or train longer. This suggests that in standard music transformers, additional model capacity is allocated to memorizing absolute patterns rather than capturing shared musical structures. In this paper, we propose the Equivariant Music Transformer (EMT), which enforces equivariance through self-distillation by jointly optimizing a next-token-prediction and an auxiliary equivariance regularization loss. We find that the additional equivariance loss acts as a beneficial regularizer, simultaneously improving next-token prediction and producing equivariant latent representations. Through both objective and subjective evaluations, EMT demonstrates superior equivariance and generative capability compared to data augmentation, feature engineering, and state-of-the-art (SOTA) baselines. More broadly, our findings reveal that standard language modeling methods alone do not capture music's translational symmetries, and dedicated inductive biases are required to produce better music representations. The code, weights and demos are available online.