Research map / Federated learning and privacy / Differential privacy
Large Language Models Can Be Strong Differentially Private Learners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a larger pretrained model | DP fine-tuning for NLP tasks · Sentence classification and language generation | task performance | Helped |
| Add text infilling to classification fine-tuning (ablation) | RoBERTa-large · SST-2 dev, ε=3 | accuracy: 90.71 -> 93.04 | Helped |
| Add text infilling to classification fine-tuning (ablation) | RoBERTa-large · QNLI dev, ε=3 | accuracy: 88.94 -> 90.76 | Helped |
| Use full fine-tuning instead of prefix-tuning (ablation) | GPT-2 · E2E test, ε=3 | BLEU: 47.772 -> 61.519 | Helped |
| Use full fine-tuning instead of training only the top 2 Transformer blocks (ablation) | GPT-2 · E2E test, ε=3 | BLEU: 25.920 -> 61.519 | Helped |
| Use pretrained initialization instead of training from scratch (ablation) | GPT-2 · E2E test, ε=3 | BLEU: 15.457 -> 61.519 | Helped |
| Increase batch size while holding update steps and other hyperparameters fixed | DP fine-tuning · E2E | task performance | Helped |
| Use a larger batch size together with a larger learning rate at fixed epochs | GPT-2 with DP-Adam · E2E test | BLEU: ~ 62 | Helped |
| Use the linear learning-rate/batch-size scaling rule in the small-batch regime | Private optimization · E2E | BLEU | Hurt |
| Use a small clipping norm that clips almost all gradients | DP optimization · Fine-tuning tasks | task performance | Helped |
| Choose label words more aligned with the classification task (ablation) | SST-2 text-infilling classifier · SST-2 | accuracy: 1-2% | Helped |
| Reduce update dimensionality with parameter-efficient adaptation (ablation) | DP fine-tuning · Sentence classification and generation | task performance | Mixed |
Closest papers by meaning
- DPZero: Private Fine-Tuning of Language Models without Backpropagation (ICML 2024 · similarity 0.91)
- DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineer (ICLR 2024 · similarity 0.90)
- Scaling Laws for Differentially Private Language Models (ICML 2025 · similarity 0.90)
- Privacy-Preserving In-Context Learning for Large Language Models (ICLR 2024 · similarity 0.89)
- PrE-Text: Training Language Models on Private Federated Data in the Age of LLMs (ICML 2024 · similarity 0.89)
- DR-Encoder: Encode Low-rank Gradients with Random Prior for Large Language Models Differentially Privately (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.