Research map / Federated learning and privacy / Differential privacy
PE-SGD: Differentially Private Deep Learning via Evolution of Gradient Subspace for Text
Tianyuan Zou, Zinan Lin, Sivakanth Gopi, Yang Liu, Ya-Qin Zhang, Robert Sim, Xin Deng, Sergey Yekhanin
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace DP-SGD with PE-SGD at eps=1 | Qwen2.5-3B-Instruct · Congressional Speech | next-token loss: 3.1080 -> 2.7532 | Improved |
| replace DP-SGD with PE-SGD at eps=1 | Qwen2.5-3B-Instruct · PubMed | next-token loss: 2.3731 -> 2.1990 | Improved |
| evolve synthetic projection samples during training (ablation) | PE-SGD · Congressional Speech (eps=1) | next-token loss: 2.8024 -> 2.7532 | Improved |
| evolve synthetic projection samples during training (ablation) | PE-SGD · PubMed (eps=1) | next-token loss: 2.2497 -> 2.1990 | Improved |
| use PE-SGD without privacy noise | Qwen2.5-3B-Instruct · Congressional Speech (eps=inf) | next-token loss: 2.6868 -> 2.7468 | Hurt |
| add noise to final projection coefficients rather than inner products (ablation) | PE-SGD · Qwen2.5 (Fig. 8) | loss: STD/Mean 0.533 -> 0.404 | Improved |
| add noise to coefficients rather than to private gradient (ablation) | PE-SGD · Qwen2.5 (Fig. 8) | loss and accuracy: lower loss, slightly higher accuracy | Improved |
| use whole synthetic gradient subspace rather than top-k eigenspace | PDP-SGD/GEP · Congressional Speech (eps=1) | next-token loss: 2.8025 -> 2.7532 | Improved |
| generate synthetic data with GPT-4o-mini instead of Qwen2.5-3B | PE-SGD · Congressional Speech (eps=1) | next-token loss: 2.7532 -> 2.7219 | Improved |
| add synthetic-data SFT before DP training (ablation) | PE-SGD · Congressional Speech (eps=1) | next-token loss: 2.7740 -> 2.7532 | Improved |
| use extreme API fold L=1 or L=infinity (ablation) | PE-SGD · Congressional Speech | loss | Hurt |
| change prompt templates (ablation) | PE-SGD · Congressional Speech | next-token loss: 2.7532 -> 2.7526 | No clear effect |
Closest papers by meaning
- Bypassing the Ambient Dimension: Private SGD with Gradient Subspace Identification (ICLR 2021 · similarity 0.91)
- Differentially Private Stochastic Gradient Descent with Fixed-Size Minibatches: Tighter RDP Guarantees with or without Replacement (NeurIPS 2024 · similarity 0.89)
- Private Evolution Converges (NeurIPS 2025 · similarity 0.88)
- Differentially Private SGD Without Clipping Bias: An Error-Feedback Approach (ICLR 2024 · similarity 0.88)
- Differentially Private Prototypes for Imbalanced Transfer Learning (AAAI 2025 · similarity 0.88)
- Privacy without Noisy Gradients: Slicing Mechanism for Generative Model Training (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
