Research map / Federated learning and privacy / Differential privacy
When Does Differentially Private Learning Not Suffer in High Dimensions?
Xuechen Li, Daogao Liu, Tatsunori Hashimoto, Huseyin A Inan, Janardhan Kulkarni, YinTat Lee, Abhradeep Guha Thakurta
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Project clipped gradients onto top k=100 principal components during private fine-tuning (ablation) | DP-SGD on DistilRoBERTa · SST-2 dev | accuracy: 90 -> 90 (original private accuracy recovered at k=100) | No clear effect |
| Project gradients onto fewer components (k in 10, 20) rather than 100 (ablation) | DP-SGD on DistilRoBERTa · SST-2 dev | accuracy | Mixed |
| Use usual classification head (random linear head on [CLS]) instead of template-completion formulation for subspace retraining (ablation) | Projected DP-SGD d=100 · SST-2 dev | accuracy: 90 -> 50 | Hurt |
| Increase orthogonal iterations T from 10 to 100 (ablation) | Spectral analysis of gradients · DistilRoBERTa SST-2 gradients | eigenspectrum decay | No clear effect |
| Vary number of gradient samples r in PCA (ablation) | Spectral analysis of gradients · DistilRoBERTa SST-2 gradients | eigenspectrum decay | No clear effect |
| Exclude initial gradients (use steps 300-1300 instead of first 1000) (ablation) | Spectral analysis of gradients · DistilRoBERTa SST-2 gradients | eigenspectrum decay | No clear effect |
| Scale model to Roberta-base (14M) and Roberta-large (50M trainable params) | Gradient spectral analysis · SST-2 fine-tuning | eigenspectrum decay | No clear effect |
| Increase dimension d with constant Lipschitz coefficients (A_const) | DP-SGD on generalized geometric median · Synthetic geometric median | excess empirical loss | Hurt |
| Use rapidly decaying coefficients (A_sqrt, A_linear) instead of A_const | DP-SGD on generalized geometric median · Synthetic geometric median | excess empirical loss vs dimension | Improved |
Closest papers by meaning
- Certification for Differentially Private Prediction in Gradient-Based Training (ICML 2025 · similarity 0.89)
- Position: Considerations for Differentially Private Learning with Large-Scale Public Pretraining (ICML 2024 · similarity 0.88)
- Public-data Assisted Private Stochastic Optimization: Power and Limitations (NeurIPS 2024 · similarity 0.87)
- Revisiting Differentially Private ReLU Regression (NeurIPS 2024 · similarity 0.87)
- Differentially private optimization for non-decomposable objective functions (ICLR 2025 · similarity 0.87)
- Differentially Private Optimization with Sparse Gradients (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
