这篇论文用可微优化替代离散搜索，Neuron Gating和Mixed Activation找紧凑网络，MNIST上0.26M参数就达99.63%准确率，比DARTS表现更好。
该论文提出一个通用的双层优化框架，用于在MLP、CNN、RNN和Transformer等架构中搜索紧凑网络。三种可扩展方法分别基于神经元门控（NAS-NG）、混合激活（NAS-MA）及两者结合（NAS-NGMA），将离散决策转化为连续松弛。在MNIST上，NAS-NGMA达到98.68%测试准确率，MLP参数为7.69M；NAS-NG以0.26M CNN参数取得99.63%准确率。在CIFAR-10上，所提方法始终优于vanilla DARTS。实验还显示NAS-NG能优化过度参数化架构，在提升准确率的同时减少参数。
Designing Compact Neural Architectures via Neuron Gating and Mixed Activation
Neural Architecture Search (NAS) is naturally formulated as a bilevel optimization problem, where the upper-level optimizes the architecture using validation performance and the lower-level trains network parameters using training loss. However, NAS is computationally expensive due to discrete architectural decisions, exponentially growing search spaces, and the high cost of training candidate architectures. This work develops a general bilevel optimization framework for NAS across diverse architectures, including MLPs, CNNs, RNNs, and Transformers, to identify compact architectures with strong predictive performance. We propose three scalable formulations that replace discrete neuron- and activation-level decisions with continuous relaxations, enabling differentiable optimization over otherwise combinatorial architecture spaces. These formulations give rise to three NAS methods: NAS based on Neuron Gating (NAS-NG), NAS based on Mixed Activation (NAS-MA), and NAS based on Neuron Gating and Mixed Activation (NAS-NGMA). Experiments on MLPs and CNNs using MNIST and CIFAR-10 show that the proposed methods consistently identify compact architectures with competitive or improved predictive performance. On MNIST, NAS-NGMA achieves 98.68% test accuracy with 7.69M MLP parameters, while NAS-NG achieves 99.63% accuracy with only 0.26M CNN parameters. On CIFAR-10, the proposed methods consistently outperform vanilla DARTS. Further experiments demonstrate that NAS-NG can optimize substantially over-parameterized and literature-optimal architectures, improving accuracy while reducing parameters. These results establish relaxed bilevel optimization as a scalable alternative to discrete NAS and provide a general framework for efficient neuron- and activation-level architecture optimization.