Research map / Federated learning and privacy / Differential privacy
Differentially Private Representation Learning via Image Captioning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a DP image captioner (DP-Cap) from scratch on a 233M LAION subset with DP-SGD and extreme batch sizes, yielding representations that beat prior DP SOTA (ViP) on ImageNet linear probing.
Problem
Differentially private representation learning has poor privacy-utility trade-offs; prior DP models trained from scratch learn representations no better than handcrafted features, and contrastive VLM objectives are incompatible with DP-SGD.
Approach
Use image captioning (an instance-separable loss) as the DP pre-training objective on internet-scale image-text data, combined with synthetic (Shaders21K) initialization, extreme batch sizes (up to 1.3M) to reduce effective noise, and ghost norm plus mixed precision for efficient DP-SGD.
Key results
- Under epsilon=8 on LAION, DP-Cap linear probe reaches 65.8% on ImageNet-1K (Large model), above prior DP SOTA ViP at 56.5%.
- DP-Cap achieves 58.6% VGR and 82.4% VGA on ARO, close to non-private Cap (59.9%, 87.2%) and above non-private CLIP.
- Ghost norm plus AMP gives 4.7x speedup over functorch for the Large model, reducing training from 61 to 13 days.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train image representations with DP captioning instead of DP masked autoencoding | DP-Cap vs ViP · ImageNet-1K full linear probing | accuracy: 56.5% -> 63.4% | Helped |
| increase synthetic initialization training by adding Shaders21K initialization (ablation) | DP-Cap · ImageNet-1K linear probing | accuracy: more than 24% | Helped |
| increase dataset size from 23M to 233M (ablation) | DP-Cap-Base · ImageNet-1K 10-shot | accuracy: 13.3% -> 31.8% | Helped |
| increase dataset size from 2.3M to 233M (ablation) | DP-Cap-Base · ImageNet-1K 10-shot | accuracy: 8.1% -> 31.8% | Helped |
| reduce privacy budget from ε=8 to ε=1 (ablation) | DP-Cap-Base · ImageNet-1K 10-shot | accuracy: 31.8% -> 19.9% | Hurt |
| reduce privacy budget from ε=8 to ε=1 (ablation) | DP-Cap-Base · ARO VGR | accuracy: 58.6% -> 58.3% | No clear effect |
| increase model size from Tiny to Large (ablation) | DP-Cap · ImageNet-1K 10-shot | accuracy: 25.2% -> 34.0% | Helped |
| increase batch size from 98k to 1.3M (ablation) | DP-Cap-Base · ImageNet-1K 10-shot | accuracy: 17.2% -> 31.8% | Helped |
| increase batch size from 98k to 1.3M (ablation) | ViP · ImageNet-1K full linear probing | accuracy: 53.4% -> 56.5% | Helped |
| fine-tune the decoder for one epoch with the image encoder frozen (ablation) | DP-Cap · MS-COCO test | CIDEr: 15.7 -> 51.3 | Helped |
| fine-tune the decoder for one epoch with the image encoder frozen (ablation) | Cap · MS-COCO test | CIDEr: 29.9 -> 79.2 | Helped |
| increase simulated training budget from 32 to 64 epochs (ablation) | DP-Cap TAN simulation · ImageNet-1K 10-shot | accuracy: +5% | Helped |
Surprising findings
- Extreme batch sizes (up to 1.3M) keep loss trajectories consistent at fixed effective noise, contrary to prior DP findings that large batches hurt.
- DP-Cap's zero-shot ImageNet accuracy is only 7.8%, far below non-private Cap at 25.2%, showing limited text-generation/zero-shot ability despite strong ARO scores.
Limitations
- Zero-shot ImageNet accuracy (7.8%) and captioning CIDEr (15.7) remain far below non-private Cap.
- Scaling model size gains are modest and may deteriorate at larger sizes due to lower gradient SNR.
- Privacy notions for image+caption vs image-only models are not directly comparable.
Experimental setup
Datasets: Dedup-LAION-233M Shaders21K ImageNet-1K CIFAR-10 CIFAR-100 Places-365 Places-205 iNaturalist-2021 ARO MS-COCO 2017
Benchmarks: ImageNet-1K Places-365 Places-205 iNaturalist-2021 CIFAR10 CIFAR100 ARO MS-COCO 2017
Metrics: linear probing accuracy few-shot accuracy zero-shot accuracy CIDEr
Baselines: ViP TAN DP-NFNet AlexNet SimCLR MAE CLIP Cap
Builds on: DP-SGD ViP masked autoencoder Cap Opacus
Closest papers by meaning
- ViP: A Differentially Private Foundation Model for Computer Vision (ICML 2024 · similarity 0.87)
- Differentially Private Prototypes for Imbalanced Transfer Learning (AAAI 2025 · similarity 0.87)
- Towards hyperparameter-free optimization with differential privacy (ICLR 2025 · similarity 0.86)
- Private Attribute Inference from Images with Vision-Language Models (NeurIPS 2024 · similarity 0.85)
- Differentially private optimization for non-decomposable objective functions (ICLR 2025 · similarity 0.85)
- Delving into Differentially Private Transformer (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.