Research map / Federated learning and privacy / Differential privacy
Learning Differentially Private Recurrent Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add user-level differential privacy with DP-FedAvg | LSTM language model · Reddit next-word prediction | AccuracyTop1: -0.13% | No clear effect |
| Replace a fixed-size sample with independent user sampling and the fixed-denominator estimator (ablation) | FedAvg · Reddit next-word prediction | convergence rate | No clear effect |
| Use the fixed-denominator estimator instead of the clipped-denominator estimator (ablation) | FedAvg · Reddit next-word prediction | convergence rate | No clear effect |
| Use flat clipping instead of per-layer clipping (ablation) | FedAvg · Reddit next-word prediction | convergence rate | Helped |
| Set clipping norm S in the range 10 to 20 (ablation) | FedAvg · Reddit next-word prediction | convergence rate: 10, 20 | No clear effect |
| Add Gaussian noise with sigma 0.024 to updates (ablation) | FedAvg · Reddit next-word prediction | convergence rate: 0.024 | Hurt |
| Increase expected users per round from 100 to 1250 (ablation) | FedAvg with clipping S=15 and noise sigma=0.012 · Reddit next-word prediction | accuracy: 100, 1250 | No clear effect |
| Decrease clipping and noise after 4885 rounds, switching to S=7.9 and sigma=0.0048 (ablation) | DP-FedAvg · Reddit next-word prediction | convergence: 4885, 6000, 7.9, 0.0048 | Helped |
| Add an explicit L2 penalty on each user's model updates (ablation) | FedAvg · Reddit next-word prediction | training performance | No clear effect |
| Use FedSGD instead of FedAvg (ablation) | Differentially private LSTM training · Reddit next-word prediction | iterations to equivalent accuracy | Hurt |
Closest papers by meaning
- DR-Encoder: Encode Low-rank Gradients with Random Prior for Large Language Models Differentially Privately (AAAI 2025 · similarity 0.90)
- Scaling Laws for Differentially Private Language Models (ICML 2025 · similarity 0.90)
- DP-MemArc: Differential Privacy Transfer Learning for Memory Efficient Language Models (AAAI 2025 · similarity 0.89)
- Certification for Differentially Private Prediction in Gradient-Based Training (ICML 2025 · similarity 0.88)
- Split-and-Denoise: Protect large language model inference with local differential privacy (ICML 2024 · similarity 0.88)
- Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.