Research map / Federated learning and privacy / Differential privacy

ViP: A Differentially Private Foundation Model for Computer Vision

ICML 2024 · Federated learning and privacy › Differential privacy · approach: Differentially private synthetic data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains a differentially private masked autoencoder ViT (ViP) on LAION233M with DP-AdamW at epsilon=8, warm-started from synthetic Shaders21k pre-training, and evaluates non-private transfer on vision benchmarks.

Problem

Training foundation vision models on uncurated internet-scale data raises privacy and copyright risks, but DP-SGD training of representation learners at scale is hard due to injected noise, non-per-sample-decomposable SSL objectives, and large epoch counts that inflate epsilon.

Approach

Use masked autoencoders, whose per-sample reconstruction loss is instance-separable and needs no batch normalization, so per-sample gradient clipping works; warm-start with non-private MAE pre-training on programmatically generated synthetic textures (Shaders21k) to cut the number of private epochs needed, then run DP-AdamW on LAION233M.

Key results

• ViP (epsilon=8) reaches 55.7% linear probing top-1 on ImageNet-1K, comparable to end-to-end AlexNet (56.5%) and above prior DP models TAN (49.0%) and DP-NFNet (45.3%).
• DP fine-tuning of ViP on ImageNet-1K at (8, 8e-7)-DP reaches 50.3% top-1, over 10 points above TAN's 39.2%.
• ViP achieves 45.2 AP box on COCO, exceeding the Mask R-CNN baseline of 40.0 and SimCLR's 37.9 in the appendix table.

Measured effects reported in the paper

Surprising findings

• Synthetic-only MAE pre-training (Syn)-ViP already outperforms prior SOTA DP-trained models on linear probing.
• Under the recipe, larger ViT models converge faster despite larger DP noise, and model scaling improves downstream accuracy, unlike random initialization.

Limitations

• Still falls short of non-private SimCLR on few-shot fine-tuning despite >100x more data.
• Training losses and results depend on large batch sizes (98,304) and carefully chosen noise; small batches degrade utility sharply.
• Non-private linear probing and fine-tuning are used for downstream evaluation, so the DP guarantee covers only pre-training data.

Experimental setup

Datasets: LAION400M (LAION233M subset) Shaders21k synthetic images ImageNet-1K Places-365 Places-205 iNaturalist-2021 iNaturalist-2018 CIFAR-100 Caltech-101 FGVC Aircraft MS-COCO ADE20K

Benchmarks: ImageNet-1K Places-365 Places-205 iNaturalist-2021 Aircraft Caltech-101 CIFAR-100 MS-COCO ADE20K

Metrics: top-1 accuracy average precision (AP box/mask) mIoU

Baselines: TAN DP-NFNet AlexNet SimCLR Mask R-CNN DP-ResNet-18 non-private MAE RefineNet

Builds on: DP-SGD masked autoencoders Shaders21k opacus functorch

Closest papers by meaning

• Differentially Private Representation Learning via Image Captioning (ICML 2024 · similarity 0.87)
• Differentially Private Prototypes for Imbalanced Transfer Learning (AAAI 2025 · similarity 0.87)
• Differentially private optimization for non-decomposable objective functions (ICLR 2025 · similarity 0.87)
• Delving into Differentially Private Transformer (ICML 2024 · similarity 0.86)
• Differentially Private Bias-Term Fine-tuning of Foundation Models (ICML 2024 · similarity 0.86)
• Towards hyperparameter-free optimization with differential privacy (ICLR 2025 · similarity 0.85)