Research map / Federated learning and privacy / Differential privacy
DP-AdamBC: Your DP-Adam Is Actually DP-SGD (Unless You Apply Bias Correction)
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that DP noise biases Adam's second-moment estimate so DP-Adam behaves like DP-SGD with momentum, and proposes DP-AdamBC, which subtracts the known noise bias (sigma C/B)^2 from the second moment to recover Adam-like behavior.
Problem
DP-Adam often underperforms under differential privacy, especially on vision and some NLP tasks, because DP noise adds a constant bias to Adam's second-moment estimate, breaking its sign-descent-like behavior.
Approach
Analytically show the DP noise adds a known constant bias Phi=(sigma C/B)^2 to the second moment; under typical DP parameters DP-Adam reduces to DP-SGD with momentum with a decaying learning rate. DP-AdamBC removes this known bias from the second-moment estimate (with a max-floor stability constant) at no extra privacy cost as it is post-processing of the privatized gradients.
Key results
- On SNLI at epsilon≈7, DP-AdamBC reaches 56.08% vs 52.63% for DP-Adam and 51.04% for DP-SGD (up to 3.5 points).
- On QNLI at epsilon≈7, DP-AdamBC gets 62.83% vs 61.23% for DP-Adam.
- On CIFAR10, DP-AdamBC (63.43% at epsilon≈7) is not better than DP-SGD (65.30%), and DP-SGD is best there.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Subtract the DP-noise bias from Adam's second-moment estimate | DP-Adam · SNLI, ε ≈ 7 | accuracy: 52.63% to 56.08% ($3.5$ percentage points) | Helped |
| Subtract the DP-noise bias from Adam's second-moment estimate | DP-Adam · QNLI, ε ≈ 7 | accuracy: 61.23% to 62.83% ($1.6$ percentage points) | Helped |
| Subtract the DP-noise bias from Adam's second-moment estimate | DP-Adam · CIFAR10, ε ≈ 7 | accuracy: 62.24% vs 63.43% | No clear effect |
| Subtract the DP-noise bias from Adam's second-moment estimate | DP-Adam · ogbn-arxiv, ε ≈ 3 and ε ≈ 6 | accuracy: 4 percentage point at ε=3, and 1.5 at ε=6 | Helped |
| Subtract the DP-noise bias from Adam's second-moment estimate | DP-Adam · ogbn-arxiv, ε ≈ 12 | accuracy: 54.02 vs 53.81 | No clear effect |
| Add a larger constant bias to the denominator (ablation) | Clipping-only DP-Adam on SNLI · SNLI | test accuracy | Hurt |
| Subtract an incorrect noise-bias value, either Φ'=1e-7 or Φ'=1e-9 instead of the true Φ (ablation) | DP-AdamBC on SNLI · SNLI | performance: Φ=2.4e-8; Φ'=1e-7; Φ'=1e-9 | Hurt |
| Vary the numerical stability constant γ' (ablation) | DP-AdamBC · SNLI | accuracy | Mixed |
| Tune γ in place of correcting for the DP-noise bias (ablation) | DP-Adam · SNLI | accuracy | Hurt |
| Set the moving-average coefficients too small (ablation) | DP-Adam · SNLI | performance | Hurt |
| Set the moving-average coefficients larger (ablation) | DP-Adam · SNLI | final performance | No clear effect |
| Increase the constant denominator bias (ablation) | Clipping-only Adam on SNLI · SNLI | training loss: mean squared difference of 0.015 | Mixed |
Surprising findings
- Under typical DP parameters, DP-Adam behaves essentially like DP-SGD with momentum with a specific decaying learning rate schedule.
- Correcting the second moment with a value other than the true noise bias (Phi' above or below Phi) degrades performance.
Limitations
- Bias correction gains are task-dependent; no improvement or slight loss on CIFAR10 and ogbn-arxiv at some budgets.
- Consistency analysis relies on a stationarity assumption not used in experiments.
- Correction can still produce updates with magnitude above 1 and depends on numerical stability constant tuning.
Experimental setup
Datasets: CIFAR10 SNLI QNLI ogbn-arxiv
Benchmarks: CIFAR10 SNLI QNLI ogbn-arxiv
Metrics: test accuracy
Baselines: DP-SGD DP-Adam DP2-RMSProp
Builds on: Adam DP-SGD sign descent hypothesis DP2
Closest papers by meaning
- Towards hyperparameter-free optimization with differential privacy (ICLR 2025 · similarity 0.87)
- DiSK: Differentially Private Optimizer with Simplified Kalman Filter for Noise Reduction (ICLR 2025 · similarity 0.86)
- DOPPLER: Differentially Private Optimizers with Low-pass Filter for Privacy Noise Reduction (NeurIPS 2024 · similarity 0.85)
- Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach (ICLR 2024 · similarity 0.85)
- Differentially Private Bias-Term Fine-tuning of Foundation Models (ICML 2024 · similarity 0.85)
- Differentially Private Stochastic Gradient Descent with Fixed-Size Minibatches: Tighter RDP Guarantees with or without Replacement (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.