这篇论文解决了深度神经网络中多项式激活函数的可识别性问题,对理解网络参数对称性有重要意义。
研究人员提出猜想:固定数量的不同非零多项式与足够高阶的通用多项式组合后会产生线性独立的多项式。该研究证明了两个多项式情况及多项式次数有界时的任意数量多项式情况。这一猜想与深度全连接神经网络架构的可识别性密切相关,特别是对于具有递增次数的层特定激活函数的网络架构,该研究完全刻画了产生相同端到端网络函数的参数集合。
Linear Independence of Polynomial Compositions and Identifiability of Deep Neural Networks
Motivated by theoretical problems in deep learning, we conjecture that post-composing a fixed number of pairwise distinct nonconstant polynomials with a generic polynomial of sufficiently large degree yields linearly independent polynomials. This generalizes Newman--Slater's theorem on powers of polynomials. We establish several cases of this conjecture and its origin-passing variant: We prove the result for two polynomials, and for an arbitrary number of polynomials when their degrees are bounded. Furthermore, we show how the conjecture implies a complete understanding of the identifiability (i.e., parameter symmetries) of deep fully connected neural network architectures with generic polynomial activation functions. In particular, for network architectures with layer-specific activations of increasing degree, our established versions of the conjecture fully characterize the set of parameters yielding the same end-to-end network function. As a special case, we fully resolve the identifiability of shallow polynomial networks.