Research map / Federated learning and privacy / Differential privacy
DOPPLER: Differentially Private Optimizers with Low-pass Filter for Privacy Noise Reduction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DOPPLER, a low-pass filter applied to privatized gradients in DP optimizers (DPSGD, DP-Adam, DP-GaLore) that suppresses DP noise in the frequency domain without extra privacy cost, improving test accuracy by 3-10%.
Problem
DP optimizers such as DPSGD suffer large performance degradation because injected Gaussian DP noise scales with model size and training steps, hindering DP pretraining of large models.
Approach
Treat privatized gradients across iterations as a time series; the gradient signal is low-frequency while DP noise is white, so an IIR low-pass filter applied as post-processing of the noisy gradients raises SNR at no extra privacy cost, with bias correction for initialization.
Key results
- LP-DPSGD outperforms DPSGD on MNIST, CIFAR-10, and CIFAR-100 at epsilon=8, with the paper claiming 3%-10% test accuracy gains across models and datasets.
- On GLUE with RoBERTa-base at epsilon=3, LP-DPAdamBC improves MNLI accuracy from 82.39 to 83.55 over DPAdamBC.
- Momentum-SGD is shown to be a special case of the filter, with SNR larger than vanilla DPSGD by a factor of 1.9 times under stated conditions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DOPPLER low-pass filtering to DPAdamBC | DPAdamBC with RoBERTa-base · GLUE MNLI, QQP, QNLI, and SST-2 at ε=3 and ε=8 | accuracy | Mixed |
| Add DOPPLER low-pass filtering to DPSGD | DPSGD pretraining · MNIST, CIFAR-10, and CIFAR-100 at ε=8 | test accuracy | Helped |
| Add DOPPLER low-pass filtering to DP optimizers | DPSGD, DPAdamBC, and DPGaLore · CIFAR-10 pretraining across privacy budgets | test accuracy | Helped |
| Use a second-order filter instead of a first-order filter (ablation) | LP-DPSGD · CIFAR-10 pretraining | convergence speed | Hurt |
| Use different filter coefficient choices of the same order (ablation) | LP-DPSGD · CIFAR-10 pretraining | test accuracy | Mixed |
| Use a first-order filter rather than no filter (ablation) | DPSGD · CIFAR-10 pretraining | test accuracy | Helped |
| Use automatic clipping (Norm) instead of vanilla clipping (Clip) (ablation) | LP-DPSGD · CIFAR-10 pretraining | test accuracy | Helped |
| Clip all layers as one vector (Flat) instead of clipping each layer separately (Layer) (ablation) | LP-DPSGD · CIFAR-10 pretraining | test accuracy | Helped |
| Increase the learning rate above the selected optimum (ablation) | LP-DPAdamBC · CIFAR-10 pretraining | final test accuracy | Hurt |
| Decrease the learning rate below the selected optimum (ablation) | LP-DPAdamBC · CIFAR-10 pretraining | convergence speed | Hurt |
| Add cosine-annealing learning-rate scheduling with warmup (ablation) | LP-DPSGD · CIFAR-10 pretraining, longer training | training performance | Helped |
| Add cosine-annealing learning-rate scheduling with warmup (ablation) | LP-DPSGD · CIFAR-10 pretraining, early training | convergence speed | Hurt |
Surprising findings
- On GLUE fine-tuning, DOPPLER gives only slight gains (e.g., MNLI +1.2 at epsilon=3) and the authors note no significant improvement over the originally reported DPAdam results.
Limitations
- Higher-order filters require hyperparameter tuning or prior knowledge of gradient auto-correlation, and their memory cost grows with filter order (n_a+n_b states per parameter).
- Gains are small for fine-tuning and small privacy noise settings; largest gains occur when DP noise is large.
- Theoretical analysis assumes bounded gradients (clipping inactive) and a gradient auto-correlation assumption that holds only for sufficiently small step size.
Experimental setup
Datasets: MNIST CIFAR-10 CIFAR-100 GLUE
Benchmarks: MNIST CIFAR-10 CIFAR-100 GLUE
Metrics: test accuracy power spectral density signal-to-noise ratio
Baselines: DPSGD DPAdamBC DPGaLore DPAdam
Builds on: DPSGD Adam GaLore momentum SGD automatic clipping
Closest papers by meaning
- DiSK: Differentially Private Optimizer with Simplified Kalman Filter for Noise Reduction (ICLR 2025 · similarity 0.92)
- Towards hyperparameter-free optimization with differential privacy (ICLR 2025 · similarity 0.91)
- The importance of feature preprocessing for differentially private linear optimization (ICLR 2024 · similarity 0.89)
- Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach (ICLR 2024 · similarity 0.89)
- Delving into Differentially Private Transformer (ICML 2024 · similarity 0.88)
- Differentially Private Stochastic Gradient Descent with Fixed-Size Minibatches: Tighter RDP Guarantees with or without Replacement (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.