Research map / Federated learning and privacy / Differential privacy
Differentially Private Subspace Fine-Tuning for Large Language Models
Lele Zheng, Xiang Wang, Tao Zhang, Yang Cao, Ke Cheng, Yulong Shen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| project gradients into SVD subspace and add noise only there (DP-SFT vs Full-DP) | RoBERTa-base · MNLI (eps=4) | accuracy: 0.3348 -> 0.8678 | Improved |
| DP-SFT instead of Adapter-DP | RoBERTa-base · IMDB (eps=1) | accuracy: 0.6952 -> 0.9290 | Improved |
| DP-SFT instead of LoRA-DP | RoBERTa-base · SST-2 (eps=4) | accuracy: 0.8876 -> 0.9323 | Improved |
| add DP noise (DP-SFT eps=4 vs non-private full tuning) | RoBERTa-base · SST-2 | accuracy: 0.9507 -> 0.9323 | Hurt |
| lower privacy budget from eps=4 to eps=1 for DP-SFT (ablation) | DP-SFT · MNLI | accuracy: 0.8678 -> 0.8652 | No clear effect |
| build subspace from 1 epoch of trajectory instead of 4 epochs (ablation) | DP-SFT (non-DP) · MNLI | accuracy: 0.8717 -> 0.8665 | No clear effect |
| 1-epoch subspace instead of full tuning (ablation) | DP-SFT (non-DP) · SST-2 | accuracy: 0.9507 -> 0.9438 (-0.69%) | Hurt |
| construct subspace from a different public dataset (IMDB -> SST-2) (ablation) | DP-SFT (DP eps=4) · SST-2 | accuracy: 0.9323 -> 0.9014 (-4.24% vs ideal) | Hurt |
| construct subspace from SST-2 for IMDB (ablation) | DP-SFT (DP eps=4) · IMDB | accuracy: 0.9352 -> 0.9090 (-2.98% vs ideal) | Hurt |
| build subspace with noisy DP-SGD trajectory (eps=3) instead of public non-private data (ablation) | DP-SFT · QNLI (eps_total=4) | accuracy: 0.9207 -> 0.6571 | Hurt |
| noisy-trajectory DP-SFT vs Full-DP (ablation) | DP-SFT (Noisy) · IMDB (eps_total=4) | accuracy: 0.7891 -> 0.8596 | Improved |
Closest papers by meaning
- DPZero: Private Fine-Tuning of Language Models without Backpropagation (ICML 2024 · similarity 0.90)
- Differentially Private Bias-Term Fine-tuning of Foundation Models (ICML 2024 · similarity 0.90)
- Privacy-Preserving Instructions for Aligning Large Language Models (ICML 2024 · similarity 0.88)
- Scaling Laws for Differentially Private Language Models (ICML 2025 · similarity 0.88)
- DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineer (ICLR 2024 · similarity 0.88)
- Differentially Private Federated Low Rank Adaptation Beyond Fixed-Matrix (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
