神经表征中相位的重要性:图像分类器的内部Oppenheim-Lim测试

The Importance of Phase in Neural Representations: An Internal Oppenheim-Lim Test of Image Classifiers

精选理由

这篇论文用Oppenheim-Lim实验方法测试了多个模型(ViT、CNN)的内部表示,发现相位才是关键,还解释了为什么CNN和ViT对纹理和形状的偏好不同。

AI 摘要

论文复现了Oppenheim和Lim(1981)的经典实验,在隐藏层中测试相位与幅度对图像识别的影响。在PRISM2D、GFNet和ViT-B/16中,预测完全跟随相位或符号捐赠者,删除幅度信息后准确率几乎不变。ResNet-50在ReLU后看似不遵循此模式,但ReLU前的干预显示晚期块中存在强相位编码,且DC-only控制表明读取器依赖通道式空间平均。这些架构共享相位/符号身份编码,但因整流和读取几何暴露在不同基底上,为CNN与注意力模型间的纹理-形状差距提供了机理解释。

原文 · arXiv cs.LG

The Importance of Phase in Neural Representations: An Internal Oppenheim-Lim Test of Image Classifiers

Oppenheim and Lim (1981) showed that natural images stay recognizable when reconstructed from their Fourier phase alone, while the magnitude carries little of their identity. We ask whether trained image classifiers reproduce this asymmetry inside their hidden layers, and we test it causally: given two images, we transplant the phase of one onto the magnitude of the other at a chosen layer and record which image the prediction follows. In PRISM2D, GFNet, and ViT-B/16 the prediction follows the phase or sign donor, and deleting all image-specific magnitude barely moves accuracy, so identity rides on phase while image-specific magnitude is largely dispensable to the readout. ResNet-50 at first seems to break the pattern, because transplanting sign after its ReLUs does nothing; a fair intervention before the ReLU reveals a strong latent sign code in the late blocks, and a DC-only control shows the readout consumes a channel-wise spatial average. Controls rule out the trivial case in which magnitude simply stops depending on the image. The architectures therefore share a phase/sign identity code but expose it in different bases, set by rectification and readout geometry, which gives a mechanistic account of the texture--shape gap between CNNs and attention models.