Research map / Federated learning and privacy / Differential privacy
Differentially Private Fine-tuning of Language Models
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Sergey Yekhanin, Huishuai Zhang
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace full DPSGD fine-tuning with parameter-efficient private fine-tuning | RoBERTa-Large · MNLI, SST-2, QQP, QNLI | accuracy: 86.1, 93.0, 86.7, 90.0 -> 87.8, 95.3, 87.4, 90.8 | Improved |
| freeze pre-trained weights and train LoRA parameters rather than use RGP decomposition and projection | private fine-tuning of RoBERTa-Large · SST-2 | memory: 9.1 -> 6.1 GB | Improved |
| freeze pre-trained weights and train LoRA parameters rather than use RGP decomposition and projection | private fine-tuning of RoBERTa-Large · SST-2 | seconds per epoch: 296 -> 271 | Improved |
| increase backbone size from RoBERTa-Base to RoBERTa-Large | DP LoRA fine-tuning · MNLI | accuracy: 83.5 -> 87.8 | Improved |
| increase model size from GPT-2-Medium to GPT-2-XL | DP LoRA fine-tuning · DART | BLEU: 42.0 -> 43.8 | Improved |
| increase model size from GPT-2-Large to GPT-2-XL | DP LoRA fine-tuning · E2E | BLEU: 66.7 -> 66.1 | Hurt |
| use Adapter instead of LoRA | DP fine-tuning of RoBERTa-Base · GLUE average accuracy | accuracy: 87.2 -> 87.3 | Improved |
| use LoRA instead of Adapter | DP fine-tuning of RoBERTa-Large · GLUE average accuracy | accuracy: 89.7 -> 90.3 | Improved |
| use full precision and larger batches for larger datasets | DP fine-tuning of RoBERTa-Large · MNLI | accuracy: 87.8 -> 89.0 | Improved |
| use full precision and larger batches for larger datasets | DP fine-tuning of RoBERTa-Base · GLUE average accuracy | accuracy: 87.2 -> 88.2 | Improved |
| use a larger privacy budget, from ε=1 to ε=3 | DP LoRA with RoBERTa-Large · MNLI | accuracy: 86.8 -> 87.4 | Improved |
| use a larger privacy budget, from ε=1 to ε=3 | DP LoRA with RoBERTa-Large · SST-2 | accuracy: 93.1 -> 94.1 | Improved |
Closest papers by meaning
- DPZero: Private Fine-Tuning of Language Models without Backpropagation (ICML 2024 · similarity 0.92)
- Privacy-Preserving Instructions for Aligning Large Language Models (ICML 2024 · similarity 0.91)
- DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineer (ICLR 2024 · similarity 0.90)
- Scaling Laws for Differentially Private Language Models (ICML 2025 · similarity 0.90)
- DR-Encoder: Encode Low-rank Gradients with Random Prior for Large Language Models Differentially Privately (AAAI 2025 · similarity 0.90)
- Privacy-Preserving In-Context Learning for Large Language Models (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
