Polynomial Compositions in Neural Networks
Linear Independence of Polynomial Compositions and Identifiability of Deep Neural Networks
The research provides theoretical foundations for understanding parameter symmetries in polynomial neural networks, relevant to AI model development in Asia.
The paper conjectures that composing distinct polynomials with a high-degree polynomial yields linearly independent results. The authors prove this for two polynomials and for any number with bounded degrees. This research addresses identifiability in deep neural networks with polynomial activations. For networks with increasing degree activations, the findings fully characterize parameter symmetries.
Linear Independence of Polynomial Compositions and Identifiability of Deep Neural Networks
Motivated by theoretical problems in deep learning, we conjecture that post-composing a fixed number of pairwise distinct nonconstant polynomials with a generic polynomial of sufficiently large degree yields linearly independent polynomials. This generalizes Newman--Slater's theorem on powers of polynomials. We establish several cases of this conjecture and its origin-passing variant: We prove the result for two polynomials, and for an arbitrary number of polynomials when their degrees are bounded. Furthermore, we show how the conjecture implies a complete understanding of the identifiability (i.e., parameter symmetries) of deep fully connected neural network architectures with generic polynomial activation functions. In particular, for network architectures with layer-specific activations of increasing degree, our established versions of the conjecture fully characterize the set of parameters yielding the same end-to-end network function. As a special case, we fully resolve the identifiability of shallow polynomial networks.
