Sharpness-Aware Initialization: Improving Differentially Private Machine Learning from First Principles

Recent advances in privacy-preserving machine learning underscore the critical role of differential privacy (DP) in protecting individual data. However, the noise introduced during DP training often leads to significant performance degradation, creating a major challenge for differentially private machine learning (DPML).

In this work, we address this challenge by controlling the detrimental effects of DP noise. Specifically, we focus on enhancing a model's robustness to random perturbations, thereby mitigating their negative impact on convergence—a central factor in maintaining high utility under DP. To this end, we propose sharpness-aware initialization (SAI), a method for improving the accuracy of DPML algorithms by achieving a flatter loss landscape. Our approach employs a two-phase training framework: SAI followed by standard Differentially Private Stochastic Gradient Descent (DPSGD). This strategy capitalizes on the observation that loss-landscape flatness converges more rapidly than the training loss, enabling an early stop on flatness optimization to limit divergence risk, followed by a phase dedicated to training-loss optimization. Moreover, splitting the training into two distinct phases allows for different privacy budgets in each phase, aligning their respective optimization objectives and tolerance to DP noise, which further mitigates performance degradation. Our experimental results show that SAI substantially improves the accuracy of state-of-the-art DPML algorithms across a range of datasets and model architectures, achieving gains of over 6% on CIFAR-10 under epsilon=1.