Research map / Federated learning and privacy / Differential privacy
Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiceSGD, a differentially private SGD with a clipped error-feedback mechanism that removes the constant clipping bias of DPSGD-GC while allowing a problem-independent clipping threshold, with Rényi-DP analysis.
Problem
DPSGD with gradient clipping converges only to a neighborhood (constant bias) unless clipping thresholds depend on unknown problem constants, degrading accuracy in practice.
Approach
A clipped error-feedback mechanism accumulates the unclipped error between clipped and unclipped updates, keeps it as a non-privatized hidden state, and feeds back a clipped version of it, so the fixed point of the algorithm is the true stationary point for any clipping threshold C2 >= C1 that is problem-independent.
Key results
- On CIFAR-10 with C=1.0 and (2,1e-5)-DP, DiceSGD reaches 97.4% test accuracy versus 95.2% for DPSGD-GC (non-private SGD 99.0%).
- On CIFAR-100 with C=1.0, DiceSGD achieves 86.3% versus 79.0% for DPSGD-GC.
- Fine-tuning GPT-2 on E2E NLG with (8,8e-6)-DP, DiceSGD gets BLEU 62.6 versus 56.8 for DPSGD-GC.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DiceSGD instead of DPSGD-GC with C=1.0 | DP ViT-small fine-tuning · CIFAR-10 | test accuracy: 95.2% -> 97.4% | Helped |
| Use DiceSGD instead of DPSGD-GC with C=0.1 | DP ViT-small fine-tuning · CIFAR-10 | test accuracy: 94.5% -> 97.5% | Helped |
| Use DiceSGD instead of DPSGD-GC with C=1.0 | DP ViT-small fine-tuning · CIFAR-100 | test accuracy: 79.0% -> 86.3% | Helped |
| Use DiceSGD instead of DPSGD-GC with C=0.1 | DP ViT-small fine-tuning · CIFAR-100 | test accuracy: 78.9% -> 86.5% | Helped |
| Use DiceSGD instead of DPSGD-GC | DP GPT-2 fine-tuning · E2E NLG Challenge | BLEU: 56.8 -> 62.6 | Helped |
| Use DiceSGD instead of DPSGD-GC | DP GPT-2 fine-tuning · E2E NLG Challenge | NIST: 4.83 -> 7.05 | Helped |
| Use DiceSGD instead of DPSGD-GC | DP GPT-2 fine-tuning · E2E NLG Challenge | METEOR: 36.2 -> 38.5 | Helped |
| Use DiceSGD instead of DPSGD-GC | DP GPT-2 fine-tuning · E2E NLG Challenge | ROUGE-L: 65.2 -> 66.6 | Helped |
| Use DiceSGD instead of DPSGD-GC | DP GPT-2 fine-tuning · E2E NLG Challenge | CIDEr: 1.43 -> 1.83 | Helped |
| Use a smaller gradient clipping threshold C1 (ablation) | DiceSGD · CIFAR-10 and CIFAR-100 | test accuracy | Helped |
| Set feedback clipping threshold C2 equal to gradient clipping threshold C1 (ablation) | DiceSGD · CIFAR-10 and CIFAR-100 | test accuracy | Helped |
Surprising findings
- DiceSGD performs better with smaller clipping thresholds, and C2 = C1 gave the best results in most ablation cases.
Limitations
- Convergence guarantees assume strong convexity or PL condition in the conclusion, and bounded gradient/variance assumptions in the theorems.
- DiceSGD requires somewhat larger DP noise than DPSGD (larger by a constant factor) due to the hidden error state.
- Experiments use a Adam variant and are limited to fine-tuning of ViT-small and GPT-2.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 E2E NLG Challenge ImageNet
Benchmarks: CIFAR-10 CIFAR-100 E2E NLG Challenge
Metrics: test accuracy BLEU NIST METEOR ROUGE-L CIDEr
Baselines: DPSGD-GC SGD
Builds on: DPSGD error feedback SGD Renyi differential privacy DP-Adam
Closest papers by meaning
- Clipped SGD Algorithms for Performative Prediction: Tight Bounds for Stochastic Bias and Remedies (ICML 2025 · similarity 0.90)
- Online Sensitivity Optimization in Differentially Private Learning (AAAI 2024 · similarity 0.89)
- Differentially Private Stochastic Gradient Descent with Fixed-Size Minibatches: Tighter RDP Guarantees with or without Replacement (NeurIPS 2024 · similarity 0.89)
- GeoClip: Geometry-Aware Clipping for Differentially Private SGD (NeurIPS 2025 · similarity 0.89)
- DOPPLER: Differentially Private Optimizers with Low-pass Filter for Privacy Noise Reduction (NeurIPS 2024 · similarity 0.89)
- The importance of feature preprocessing for differentially private linear optimization (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.