Research map / Federated learning and privacy / Personalized and heterogeneous FL
On the Benefits of Public Representations for Private Transfer Learning under Distribution Shift
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows empirically that CLIP-based public features improve private transfer under extreme concept shift (up to 67% over private training from scratch), and theoretically that a shared low-rank subspace lets public pretraining reduce private sample complexity, with a matching two-stage lower bound.
Problem
Whether public pretraining helps differentially private learning when public and private tasks have large distribution shift (concept shift), where zero-shot and from-scratch private training both perform poorly.
Approach
Empirically, private linear probing and finetuning on off-the-shelf public CLIP features beat private training from scratch on out-of-distribution tasks; theoretically, a two-stage method (estimate shared low-rank subspace from public tasks via method of moments, then run DP-SGD in the projected k-dimensional space) achieves rates scaling with subspace rank k rather than ambient dimension d, plus
Key results
- Private linear probing on CLIP features improves accuracy over private training from scratch by up to 67% across PCam, fMoW, and RESISC45.
- Private linear probing consistently outperforms private full finetuning; on RESISC45 it beats full finetuning by over 50% at all epsilon levels.
- Zero-shot CLIP ViT-B/32 gets 51.2% on PCam (near random) and 1.64% on fMoW, while nonprivate linear probing reaches 83.5% on PCam and 42.1% on fMoW.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use pretrained public features for private training instead of training from scratch | PCam · PCam | accuracy: up to 4% | Helped |
| Use pretrained public features for private training instead of training from scratch | fMoW · fMoW | accuracy | Helped |
| Use pretrained public features for private training instead of training from scratch | RESISC45 · RESISC45 | accuracy: up to 67% | Helped |
| Fix pretrained features and train a linear probe instead of privately finetuning the full model | Private transfer learning · PCam, fMoW, and RESISC45 | accuracy | Helped |
| Use linear probing instead of full finetuning | Private transfer learning · RESISC45 | accuracy: over 50% | Helped |
| Pretrain features on a 14M subset of Datacomp-1B instead of LAION-2B | Private linear probing · PCam, fMoW, and RESISC45 | accuracy | Hurt |
| Increase public samples for subspace estimation from n1=500 to n1=2000 | Two-phase private transfer algorithm · Simulated linear regression | parameter error | Helped |
Surprising findings
- Private linear probing outperforms private full finetuning on all three out-of-distribution datasets, sometimes by large margins.
- Features from the smaller Datacomp-1B subset yield worse linear-probe accuracy than LAION-2B features, though still above training from scratch.
Limitations
- The linear low-rank subspace model is a simplification of neural network representations used in practice.
- The upper bound relies on an oracle subspace estimate and on method-of-moments conditions (task diversity) that may not hold in practice.
- Empirical evaluation covers only three datasets with a single CLIP ViT-B/32 backbone family.
Experimental setup
Datasets: PatchCamelyon fMoW RESISC45 LAION-2B Datacomp-1B (14M subset)
Benchmarks: PatchCamelyon fMoW RESISC45 CIFAR-10
Metrics: test accuracy l2 parameter error
Baselines: Zero-shot CLIP Training from scratch (private) Full finetuning (private) Linear probing (non-private) DP-SGD without subspace estimate
Builds on: DP-SGD method-of-moments subspace estimation for meta-learning PILLAR Opacus
Closest papers by meaning
- Position: Considerations for Differentially Private Learning with Large-Scale Public Pretraining (ICML 2024 · similarity 0.88)
- Differentially Private Prototypes for Imbalanced Transfer Learning (AAAI 2025 · similarity 0.86)
- In-distribution Public Data Synthesis with Diffusion Models for Differentially Private Image Classification (CVPR 2024 · similarity 0.85)
- Optimal Differentially Private Model Training with Public Data (ICML 2024 · similarity 0.85)
- It's My Data Too: Private ML for Datasets with Multi-User Training Examples (ICML 2025 · similarity 0.84)
- Public-data Assisted Private Stochastic Optimization: Power and Limitations (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.