Research map / Federated learning and privacy / Differential privacy
DP-MemArc: Differential Privacy Transfer Learning for Memory Efficient Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DP-MemArc, two memory-efficient differentially private fine-tuning designs (side network and reversible network) combined with book-keeping clipping, reducing training memory on RoBERTa-large and GPT-2-large.
Problem
Differentially private fine-tuning of LLMs still incurs large training memory costs, and existing DP parameter-efficient methods yield only marginal memory savings.
Approach
Combine side-network (LST-style) and reversible-network (LoRA/adapter in an invertible block) designs with BK-MixOpt per-sample gradient clipping so that activations need not be stored, cutting DP training memory while retaining accuracy.
Key results
- DP-MemArc_side on RoBERTa-large reduces memory to 6.18 GB at epsilon=1.6 versus 12.65 GB for DP-LoRA, while matching accuracy (e.g., MNLI 81.30 vs 81.49).
- DP-MemArc_rev on RoBERTa-large achieves 5.48 GB at epsilon=1.6 with average accuracy about 3.1 points above DP-BiTFiT, per the paper.
- On GPT-2-large E2E, DP-MemArc_rev uses 9.45 GB at epsilon=1.6 vs 58.96 GB for DP-Full FT, with BLEU 65.1 vs 62.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DP-MemArc_side instead of DP-LoRA | RoBERTa-large · MNLI (ε=1.6) | memory: 12.65 -> 6.18 | Helped |
| Use DP-MemArc_rev instead of DP-BiTFiT | RoBERTa-large · QQP (ε=1.6) | accuracy: 78.92 -> 82.57 | Helped |
| Use DP-MemArc_side instead of DP-LoRA | GPT-2-large · E2E (ε=1.6) | BLEU: 65.4 -> 66.4 | Helped |
| Use DP-MemArc_side instead of DP-Adapters | GPT-2-large · E2E (ε=1.6) | Rouge-L: 64.9 -> 64.6 | Hurt |
| Use DP-MemArc_side instead of DP-Adapters | GPT-2-large · E2E (ε=1.6) | perplexity: 2.44 -> 2.32 | Helped |
| Use BK-MixOpt for DP-MemArc_side instead of Opcaus (ablation) | DP-MemArc_side · privacy constraint ε=1.6 | memory: 7.45 -> 6.18 | Helped |
| Use BK-MixOpt for DP-MemArc_rev instead of Opcaus (ablation) | DP-MemArc_rev · privacy constraint ε=1.6 | memory: 10.66 -> 5.48 | Helped |
| Replace LoRA in reversible-network sub-function F with another PEFT architecture (ablation) | DP-MemArc_rev · privacy constraint ε=8.0 | accuracy: +0.71 | Hurt |
Surprising findings
- DP-MemArc_side memory at epsilon=infinity (5.66 GB) is close to its private memory, and under Opacus per-sample gradient computation the reversible design uses more memory than the side design at epsilon=1.6 (10.66 vs 7.45 GB).
Limitations
- Side-network performance deteriorates at small reduction ratios r, requiring LST-style intermediate connections.
- Reversible approximation (alpha near zero) may need more training steps to converge, as noted in training-scale analysis.
- Some memory comparisons are reported without a stated hardware setup, and results are presented with limited detail on variance.
Experimental setup
Datasets: MNLI QQP QNLI SST2 E2E
Benchmarks: MNLI QQP QNLI SST2 E2E
Metrics: accuracy training memory (GB) BLEU Rouge-L perplexity time (sec)
Baselines: DP-LoRA DP-Adapters PromptDPSGD DP-BiTFiT DP-Full FT
Builds on: DP-SGD Book-Keeping Side-tuning LST LoRA Adapters
Closest papers by meaning
- Scaling Laws for Differentially Private Language Models (ICML 2025 · similarity 0.89)
- DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineer (ICLR 2024 · similarity 0.88)
- Privacy Auditing of Large Language Models (ICLR 2025 · similarity 0.87)
- Privacy-Preserving Instructions for Aligning Large Language Models (ICML 2024 · similarity 0.87)
- Towards hyperparameter-free optimization with differential privacy (ICLR 2025 · similarity 0.87)
- Split-and-Denoise: Protect large language model inference with local differential privacy (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.