想在不牺牲太多性能的前提下给贝叶斯神经网络加差分隐私？这篇论文的DP-IVON-Gradsq方法在CIFAR-10上中低隐私保护时表现不错，兼容Adam效率，值得一看。
本文提出DP-IVON-Gradsq，一种基于改进变分在线牛顿（IVON）优化器的差分隐私训练方法。它通过噪声校正的平方梯度估计器从私有化梯度构建曲率估计，减少后验采样噪声与隐私噪声的交互，同时保持IVON类似Adam的计算效率。在CIFAR-10上与DP-SGD和DP-Adam对比，在弱到中等隐私预算（ε较大）下表现具有竞争力，但在强隐私约束下性能下降。代码已开源。
DP-IVON-Gradsq: Differentially Private Squared-Gradient Improved Variational Online Newton
Differential privacy provides formal privacy guarantees for training neural networks on sensitive data, while Bayesian deep learning offers a principled framework for uncertainty-aware prediction. Combining these two objectives remains challenging, as privacy noise can interact with the stochasticity introduced by Bayesian posterior sampling. In this work, we investigate differentially private variational Bayesian learning through the Improved Variational Online Newton (IVON) optimizer. We introduce DP-IVON-Gradsq, a private variant of IVON. The proposed method constructs its curvature estimate from the privatized gradient using a noise-corrected squared-gradient estimator, reducing the direct interaction between posterior-sampling noise and privacy noise while preserving the Adam-like computational efficiency of IVON. We evaluate DP-IVON-Gradsq on CIFAR-10 against the standard private optimizers DP-SGD and DP-Adam over a range of privacy budgets. The results show that DP-IVON-Gradsq is competitive under weak-to-moderate privacy constraints, i.e., large-to-moderate values of $\varepsilon$, while degrading under strong privacy. Code is available at https://github.com/NourJamoussi/DP-IVON-Gradsq.git.