Two-Layer Quantized Neural Networks Stability
Stability and Generalization of Straight-Through Estimators for Training Two-Layer Quantized Neural Networks
Theoretical framework explains why discontinuous STE training works for quantized neural networks, with practical implications for binary neural network implementations.
The paper analyzes identity straight-through estimators (STE) for training two-layer binary-activation networks with hinge loss.
Researchers derive exact distance identities for coupled updates and prove approximate non-expansiveness of the common-example map.
The study provides explicit ℓ₂ on-average model-stability and generalization bounds, with an O(n⁻¹/²) rate when T=n².
Stability and Generalization of Straight-Through Estimators for Training Two-Layer Quantized Neural Networks
We study the identity straight-through estimator (STE) for training a two-layer binary-activation network with hinge loss from the perspective of Statistical Learning Theory (SLT). Our central question is whether algorithmic stability can explain the statistical generalization of the estimator produced by the discontinuous STE training rule. In the saturated-output regime, the zero-initialized samplewise STE recursion is exactly the stochastic subgradient descent on the convex latent loss $(-yu^\top x)_+$. This representation makes a stability analysis possible. We derive an exact distance identity for two coupled updates and prove approximate non-expansiveness of the common-example map, with a quadratic defect only when the two latent margins straddle zero. We then obtain explicit $\ell_2$ on-average model-stability and generalization bounds, transferring stability isometrically from the latent vector to the full first-layer matrix. Combining stability with a standard optimization bound yields an explicit excess induced-risk guarantee and the rate $O(n^{-1/2})$ when $T=n^2$. Under margin separability, a complementary argument gives the optimal-order $O(R^2/(γ^2n))$ expected excess misclassification error for a randomized one-pass STE iterate and a corresponding majority-vote bound.
This report is categorized under AITOP's "Papers" desk. Within our cross-regional analytical framework, it directly maps to AI developer toolchains, coding agents, and engineering workflows. Key entities and technical topics under continuous tracking include STE, statistical learning theory.
Source attribution: curated from arXiv cs.LG (@Yiming Ying).
