Research map / Federated learning and privacy / Differential privacy
Pre-training Differentially Private Models with Limited Public Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a Hessian-based analysis of DP training slowdown and a two-phase DP continual pre-training strategy that first pre-trains publicly on ImageNet-1k (~10%) then privately on ImageNet-11k, yielding DP ViT-Base models.
Problem
DP pre-training of large foundation models suffers severe utility degradation from DP noise, so most DP work is limited to fine-tuning and a large share of pre-training data remains unprotected.
Approach
Analyze per-iteration loss improvement via the Hessian to show DP noise slowdown (decelerator sigma^2 tr(H)/(Bc^2)) is large during pre-training but small after public pre-training; hence a short public pre-training phase followed by DP continual pre-training closes most of the utility gap.
Key results
- Using ~10% public data, DP continual pre-training reaches 41.5% upstream accuracy on ImageNet-11k at epsilon=8 (39.8% at epsilon=2).
- Non-DP linear/full-parameter transfer reaches 55.7% on Places365 and 60.0% on iNat2021 (vs ViP 46.1% and 38.1%), with only 0.7B images processed.
- Downstream fine-tuning: DP model achieves 98.4% CIFAR10, 90.2% CIFAR100, 86.5% Food101, 96.8% SVHN (non-DP fine-tuning), improving over DINO by 0.3-2.0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Continue DINO pre-training privately on ImageNet-11k using limited public ImageNet-1k data | ViT-Base pre-training · CIFAR100 (standard fine-tuning) | accuracy: 88.2 -> 90.2 | Helped |
| Continue DINO pre-training privately on ImageNet-11k using limited public ImageNet-1k data | ViT-Base pre-training · CIFAR10 (standard fine-tuning) | accuracy: 98.1 -> 98.4 | Mixed |
| Continue DINO pre-training privately on ImageNet-11k using limited public ImageNet-1k data | ViT-Base pre-training · Food101 (standard fine-tuning) | accuracy: 85.2 -> 86.5 | Helped |
| Continue DINO pre-training privately on ImageNet-11k using limited public ImageNet-1k data | ViT-Base pre-training · SVHN (standard fine-tuning) | accuracy: 96.2 -> 96.8 | Helped |
| Use ε=8 rather than ε=2 for DP pre-training | ViT-Base continual pre-training · ImageNet-11k | accuracy: 39.8% -> 41.5% | Helped |
| Use DP continual pre-training rather than DINO initialization alone | ViT-Base · Places365 (full fine-tuning) | accuracy: 52.1 -> 55.7 | Helped |
| Use DP continual pre-training rather than DINO initialization alone | ViT-Base · iNat2021 (full fine-tuning) | accuracy: 43.5 -> 60.0 | Helped |
| Add a limited public-data pre-training phase before private training | DP pre-training · Pre-training | loss improvement | Helped |
| Use public and private data together rather than public data alone | Mixed data training · Pre-training | loss improvement | Helped |
| Re-initialize the first-order AdamW momentum when switching from public to private training (ablation) | ViT-Base AdamW · CIFAR100 | performance | Helped |
| Re-initialize AdamW states other than first-order momentum, or do not re-initialize states, at the public-to-private switch (ablation) | ViT-Base AdamW · CIFAR100 | performance | Hurt |
Surprising findings
- Increasing batch size does not remove DP pre-training slowdown because the loss-improvement denominator term B G^T H G grows with B.
- Analysis suggests DP-SGD with optimal batch size is as fast as non-DP SGD with twice the batch size.
Limitations
- Mixed-data training (optimal alpha) is approximated with binary public/private switching, not the full mixing analyzed theoretically.
- Membership inference evaluation is a single white-box logistic-regression attack on ImageNet-11k, a limited privacy evaluation.
- Relies on approximation c ~ E[C_i] and Taylor approximations; Hessian trace estimated with Hutchinson sampling.
Experimental setup
Datasets: ImageNet-1k ImageNet-11k ImageNet-21k CIFAR10 CIFAR100 Food101 SVHN Aircraft Places365 iNat2021 CodeParrot E2E
Benchmarks: ImageNet-11k CIFAR10 CIFAR100 Food101 SVHN Aircraft Places365 iNat2021 ImageNet-1k
Metrics: accuracy membership inference attack accuracy/AUC BLEU
Baselines: DINO MIIL ViP TAN AugReg Original ViT
Builds on: DINO MIIL DP-SGD BiTFiT DPMD
Closest papers by meaning
- Optimal Differentially Private Model Training with Public Data (ICML 2024 · similarity 0.88)
- Differentially Private Synthetic Data via Foundation Model APIs 1: Images (ICLR 2024 · similarity 0.86)
- Differentially Private Bias-Term Fine-tuning of Foundation Models (ICML 2024 · similarity 0.86)
- Towards hyperparameter-free optimization with differential privacy (ICLR 2025 · similarity 0.86)
- Noise-Aware Differentially Private Regression via Meta-Learning (NeurIPS 2024 · similarity 0.86)
- DiSK: Differentially Private Optimizer with Simplified Kalman Filter for Noise Reduction (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.