Research map / Federated learning and privacy / Differential privacy
Position: Considerations for Differentially Private Learning with Large-Scale Public Pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Position paper arguing that public web-scraped pretraining for differentially private learning is not truly privacy-preserving, that current benchmarks conflate public and private distributions, and that large pretrained models force private data to be outsourced.
Problem
Whether public-pretraining-then-private-finetuning is a valid and useful route to differentially private learning, given that web-scraped pretraining data may be sensitive and current benchmarks overlap heavily in distribution.
Approach
Critically reviews the public-pretrain/private-finetune paradigm along three axes: privacy semantics of public web data, benchmark validity (public and private distributions overlap), and the privacy cost of outsourcing very large models. No new method or experiments are presented.
Key results
- CLIP reaches 96.2% zero-shot accuracy on CIFAR-10 without seeing CIFAR-10 training data, which the authors argue makes that benchmark a weak measure of private learning.
- State-of-the-art differentially private ImageNet models have over 250 million parameters and require over 100 billion FLOPs to evaluate.
- MobileBERT (about 25M parameters) is two to four orders of magnitude smaller than the language models used in private finetuning work.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use public pretraining on 4 billion web images before DP fine-tuning instead of training from scratch | Differentially private ImageNet learning · ImageNet | top-1 accuracy: 39.2\% -> 86.8\% | Helped |
| Use public pretraining on 4 billion web images before DP fine-tuning instead of training from scratch | Differentially private ImageNet learning · ImageNet | top-1 accuracy: 39.39\% -> 86.8\% | Helped |
| Use web-pretrained BASIC-L for zero-shot transfer to a medical image task | BASIC-L · PCam | top-1 accuracy: 59.6\% | Hurt |
| Use web-pretrained CLIP without fine-tuning | CLIP · CIFAR-10 | zero-shot accuracy: 96.2\% | Helped |
| Use a pretrained foundation model for zero-shot evaluation | Foundation model · ImageNet | zero-shot top-1 accuracy: 85.7\% | Helped |
Surprising findings
- Some private benchmarks are more accessible than their public pretraining data: JFT is proprietary while ImageNet is public.
- CIFAR-10 classes are all contained in ImageNet, so private finetuning there may be mostly intra-domain transfer.
Limitations
- Position paper with no new experiments; claims rest on argument and examples.
- Does not technically explore or propose solutions to the identified issues.
Experimental setup
Builds on: DP-SGD public pretraining for private learning
Closest papers by meaning
- Certification for Differentially Private Prediction in Gradient-Based Training (ICML 2025 · similarity 0.89)
- Differentially Private Prototypes for Imbalanced Transfer Learning (AAAI 2025 · similarity 0.88)
- Optimal Differentially Private Model Training with Public Data (ICML 2024 · similarity 0.88)
- Differentially Private Model Merging (NeurIPS 2026 · similarity 0.88)
- On the Benefits of Public Representations for Private Transfer Learning under Distribution Shift (NeurIPS 2024 · similarity 0.88)
- Provable Privacy with Non-Private Pre-Processing (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.