Research map / Federated learning and privacy / Differential privacy
Differentially Private Bias-Term Fine-tuning of Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DP-BiTFiT, which applies differential privacy to fine-tuning only the bias terms of pre-trained models, matching DP full fine-tuning accuracy while being much faster and more memory-efficient.
Problem
Differentially private fine-tuning of large pre-trained models is accurate but computationally and memory expensive, and parameter-efficient DP methods like LoRA and Adapter are architecture-dependent and costly in activation memory.
Approach
Train only the bias terms under DP-SGD (model-agnostic, ~0.1% of parameters). Because bias gradients need no input activations, the forward hooks that cache activations can be dropped, making per-sample gradient norm computation independent of feature dimension T.
Key results
- On RoBERTa-large under epsilon=8, DP-BiTFiT reaches 88.3% MNLI-m vs 87.0% for DP full fine-tuning, and 94.5% SST2 vs 93.8%.
- DP-BiTFiT on GPT2-large (E2E, epsilon=8) reaches BLEU 65.21 vs 64.64 for DP full fine-tuning, while training only 0.066% of parameters.
- On CIFAR10 with ViT-large at epsilon=2, DP-BiTFiT reaches 99.0% accuracy; the paper reports 2-30x speedup and 2-8x memory savings over DP full fine-tuning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train bias terms rather than all parameters with DP | RoBERTa-base · SST2 | accuracy: 92.1 -> 92.4 | Helped |
| Train bias terms rather than all parameters with DP | RoBERTa-base · QNLI | accuracy: 87.9 -> 86.9 | Hurt |
| Train bias terms rather than all parameters with DP | RoBERTa-large · MNLI-m | accuracy: 87.0 -> 88.3 | Helped |
| Train bias terms rather than all parameters with DP | GPT2-small · E2E | BLEU: 63.60 -> 60.56 | Hurt |
| Train bias terms rather than all parameters with DP | GPT2-large · E2E | BLEU: 64.64 -> 65.21 | Helped |
| Train bias terms rather than all parameters with DP | ViT-large · CIFAR100, ε=2 | accuracy: 90.1 -> 91.2 | Helped |
| Add bias terms to layers before applying DP-BiTFiT | ResNet18 · CelebA [Multi-label] | accuracy: 86.9% -> 87.3% | Helped |
| Freeze all weights while retaining forward hooks (ablation) | GhostClip on RoBERTa-large · QQP | training time per epoch: 119 minutes -> 80 minutes | Helped |
| Remove forward hooks after freezing weights (ablation) | RoBERTa-large bias-only training · QQP | training time per epoch: 80 minutes -> 63 minutes | Helped |
| Use the maximum batch size enabled by DP-BiTFiT memory savings (ablation) | RoBERTa-large DP-BiTFiT · QQP | training time per epoch: 63 minutes -> 43 minutes | Helped |
| Use one DP full fine-tuning epoch before BiTFiT instead of zero (ablation) | ViT-large · CIFAR10, ε=2 | accuracy: 90.6 -> 98.8 | Helped |
| Use one DP full fine-tuning epoch before BiTFiT instead of zero (ablation) | ViT-large · CIFAR100, ε=2 | accuracy: 19.4 -> 82.4 | Helped |
Surprising findings
- DP-BiTFiT can outperform DP full fine-tuning on several settings (e.g., GPT2-large on E2E BLEU and RoBERTa-large on MNLI), despite training only about 0.1% of parameters.
- Bias-term training fails on CNNs with batch normalization because convolution layers often have no bias, so DP-BiTFiT needs architecture modification or two-phase training.
Limitations
- Bias-only training is not directly applicable to architectures without bias terms (e.g., LLaMA, BN-based CNNs), requiring DP-BiTFiT-Add or architecture changes.
- On CNNs with batch normalization, DP-BiTFiT underperforms unless convolution biases are added or two-phase training is used.
- Accuracy gaps to DP full fine-tuning remain on some tasks such as CelebA with ResNet18 (88.17% vs 91.02% for smiling at epsilon=8).
Experimental setup
Datasets: MNLI QQP QNLI SST2 E2E CIFAR10 CIFAR100 CelebA ImageNet
Benchmarks: MNLI-m QQP QNLI SST2 E2E CIFAR10 CIFAR100 CelebA
Metrics: accuracy BLEU ROUGE-L NIST METEOR CIDEr perplexity training time memory throughput
Baselines: DP full fine-tuning (GhostClip) DP LoRA DP Adapter Compacter RGP prefix tuning DP last-layer training MixGhostClip DP full fine-tuning
Builds on: BiTFiT DP-SGD GhostClip MixGhostClip LoRA
Closest papers by meaning
- Towards hyperparameter-free optimization with differential privacy (ICLR 2025 · similarity 0.89)
- DPZero: Private Fine-Tuning of Language Models without Backpropagation (ICML 2024 · similarity 0.88)
- Differentially private optimization for non-decomposable objective functions (ICLR 2025 · similarity 0.87)
- DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineer (ICLR 2024 · similarity 0.87)
- DOPPLER: Differentially Private Optimizers with Low-pass Filter for Privacy Noise Reduction (NeurIPS 2024 · similarity 0.86)
- DiSK: Differentially Private Optimizer with Simplified Kalman Filter for Noise Reduction (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.