Research map / Federated learning and privacy / Differential privacy
DP-SGD refinements
Make private stochastic gradient descent cheaper and more practical, without sampling or at scale.
60 accepted papers at ICML, NeurIPS, ICLR, CVPR and AAAI (2018–2026) · an approach in Differential privacy, keyword cluster “dp sgd · gradient · clipping”
State History Tried Baselines Checklist Preprints Who Export
Where it stands
Not enough benchmark data Fewer than four papers report the same benchmark, so there is no record to follow.
25 papers in 2023–24 and 22 in 2025–26 (at ICML, NeurIPS, ICLR, CVPR and AAAI); its share of the topic shrank, from 11.6% to 8.3%.
Open questions the data points to
- Papers report DP-SGD on StackOverflow (accuracy) anywhere from 0.28 to 0.33: comparisons against it depend on whose number you take.
- Papers report PDA-DPMD on IMDB (accuracy) anywhere from 0.62 to 0.703: comparisons against it depend on whose number you take.
How this is decided
A benchmark is tracked when at least four papers of this approach report their own method on the same dataset and metric in at least two different years. Its record is the best number reported up to 2024 compared with the best reported in 2025–26: still improving means at least 1% better, flat means at most 0.2% better with at least two papers from 2025–26 trying. Numbers on a different scale from the rest (a fraction among percentages) and far outliers are left out. Results come from different papers with different setups, so they are signals, not head-to-head comparisons. The share of papers is counted at ICML, NeurIPS, ICLR, CVPR and AAAI.
How it got here
Most cited paper of each year
- 2018 · cpSGD: Communication-efficient and differentially-private distributed SGD (NeurIPS 2018 · 184 citations)
- 2020 · Understanding Gradient Clipping in Private SGD: A Geometric Perspective (NeurIPS 2020 · 60 citations)
- 2021 · Practical and Private (Deep) Learning Without Sampling or Shuffling (ICML 2021 · 21 citations)
- 2022 · The Role of Adaptive Optimizers for Honest Private Hyperparameter Selection (AAAI 2022 · 14 citations)
- 2023 · Differentially Private Learning with Per-Sample Adaptive Clipping (AAAI 2023 · 22 citations)
- 2024 · DP-AdamBC: Your DP-Adam Is Actually DP-SGD (Unless You Apply Bias Correction) (AAAI 2024 · 12 citations)
- 2025 · Differentially Private Prototypes for Imbalanced Transfer Learning (AAAI 2025 · 1 citation)
- 2026 · An Improved Privacy and Utility Analysis of Differentially Private SGD with Bounded Domain and Smooth Losses (AAAI 2026 · 1 citation)
Earliest papers here
- cpSGD: Communication-efficient and differentially-private distributed SGD (NeurIPS 2018 · 184 citations)
- Understanding Gradient Clipping in Private SGD: A Geometric Perspective (NeurIPS 2020 · 60 citations)
- Privacy Amplification via Random Check-Ins (NeurIPS 2020 · 2 citations)
What has been tried
45 papers here report 421 measured changes (a component added, removed or swapped, compared with a baseline): helped 65%, no clear effect 10%, hurt 19%, mixed 6%. Papers mostly publish changes that helped, so "helped" is over-represented.
Ablations in recent papers
| What was changed | Measured on | Result | Effect | Paper |
|---|---|---|---|---|
| Add a larger constant bias to the denominator | SNLI · test accuracy | Hurt | DP-AdamBC: Your DP-Adam Is Actually DP-SGD (Unless You Apply Bias Correction) (AAAI 2024 · 12 citations) | |
| Increase pre-training from 0 to 5 epochs to craft initial parameters | MNIST, ε=10.0 · ε_emp | 3.41 -> 6.48 | Helped | Nearly Tight Black-Box Auditing of Differentially Private Machine Learning (NeurIPS 2024 · 2 citations) |
| add a projection layer | CIFAR100, non-private · accuracy | 74.0% -> 72.6% | Hurt | Differentially Private Prototypes for Imbalanced Transfer Learning (AAAI 2025 · 1 citation) |
| Vary γ from 0.2 to 0.6 to 1.0 | USPS · test accuracy (%) | 85.537; 86.182; 85.752 | No clear effect | GeoClip: Geometry-Aware Clipping for Differentially Private SGD (NeurIPS 2025 · 1 citation) |
| use a linear fit with 3 points rather than 2 | ImageNet · mean accuracy | 86.69 (0.86) -> 87.81 (0.86) | Helped | A New Linear Scaling Rule for Private Adaptive Hyperparameter Optimization (ICML 2024 · 1 citation) |
| Increase the number of random directions q | Tasks in the hyperparameter study · accuracy | Helped | Private Zeroth-Order Optimization with Public Data (NeurIPS 2025 · 0 citations) | |
| increase softmax temperature from 0.01 to 0.22 | CIFAR-10, r=0/255 · certifiable accuracy | 47.4 -> 44.4 | Hurt | DP-SGD Without Clipping: The Lipschitz Neural Network Way (ICLR 2024 · 0 citations) |
| Use a smaller gradient clipping threshold C1 | CIFAR-10 and CIFAR-100 · test accuracy | Helped | Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach (ICLR 2024 · 0 citations) | |
| Use a second-order filter instead of a first-order filter | CIFAR-10 pretraining · convergence speed | Hurt | DOPPLER: Differentially Private Optimizers with Low-pass Filter for Privacy Noise Reduction (NeurIPS 2024 · 0 citations) | |
| Restrict the strategy to banded Toeplitz matrices rather than arbitrary banded strategies | Prefix workload; large n ≥ 16384 and small b ≤ 32 · RMSE suboptimality | ≤ 0.25% | Hurt | Scaling up the Banded Matrix Factorization Mechanism for Large Scale Differentially Private ML (ICLR 2025) |
Baseline numbers that disagree
The same baseline on the same benchmark and metric, reported more than 5% apart by different papers here. Training budgets, data splits and reimplementations differ, so state which number you compare with.
| Baseline | Benchmark | Metric | Reported | Papers |
|---|---|---|---|---|
| DP-SGD | StackOverflow | accuracy | 0.28 – 0.33 | 2 papers, e.g. ICML 2022 and ICLR 2023 |
| PDA-DPMD | IMDB | accuracy | 0.62 – 0.703 | 2 papers, e.g. ICML 2022 and ICLR 2023 |
| PDA-DPMD | StackOverflow | accuracy | 0.33 – 0.353 | 2 papers, e.g. ICML 2022 and ICLR 2023 |
What reviewers will expect
Datasets and baselines named in the 11 papers of this approach from 2025–2026 that have a full-text summary. Leaving out the common ones is a frequent review question.
Datasets
- CIFAR-10 · 4 of 11 papers
- CIFAR100 · 2 of 11 papers
- ImageNet-1K · 2 of 11 papers
- MNIST · 2 of 11 papers
Baselines
- DP-SGD · 5 of 11 papers
Recent preprints
Loading recent arXiv preprints…
Who publishes on it
Institutions on the most papers of this approach since 2023 (of 47 papers; from OpenAlex and the papers' affiliations).
- Google · 6 papers
- Amazon · 4 papers
- University of Minnesota, Minneapolis · 4 papers
- Meta · 4 papers
- UC Santa Barbara · 3 papers
- Princeton University · 3 papers
- Inria · 2 papers
- Google DeepMind · 2 papers
- Carnegie Mellon University · 2 papers
- University of Southern California · 2 papers
Use it
Download BibTeX of 37 key papersOpen the topic in the interactive map
Get alerts. Describe your idea in scime atlas and save it (free scime account): you get a weekly email when new preprints land close to it.
Related approaches
Same keyword cluster
- Private training with public data (96 papers)
- Differentially private synthetic data (81 papers)
- Differentially private federated learning (35 papers)
Elsewhere in the topic
- Privacy accounting and mechanisms (127 papers)
- Private online and user-level learning (116 papers)
- Local-DP and private estimation (113 papers)
- Private data analysis frameworks (105 papers)
- Differentially private clustering (41 papers)
Benchmark numbers, ablations and baseline numbers are extracted by AI from each paper's full text and can be wrong; the paper is the authority. Data: results index of 2026-11-02, citation counts from OpenAlex (2026-10-09).