Research map / Federated learning and privacy / Differential privacy
Differentially Private Prototypes for Imbalanced Transfer Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Differentially Private Prototype Learning (DPPL), which builds per-class DP prototypes from frozen public pre-trained encoder features, either via DP means (DPPL-Mean) or by privately selecting public prototypes with the exponential mechanism (DPPL-Public), targeting high-privacy, low-data, and imbalanced transfer learning.
Problem
Differentially private transfer learning via DP-SGD degrades badly in high privacy (small epsilon), low-data, and class-imbalanced settings.
Approach
Replace iterative DP-SGD training with one-shot per-class DP prototypes on frozen public encoder features; DPPL-Mean releases a Gaussian-noised class mean (zCDP), while DPPL-Public privately selects a public data point per class via the exponential mechanism (pure epsilon-DP), exploiting public data beyond encoder pre-training.
Key results
- DPPL-Public strongly outperforms DP linear probing on imbalanced CIFAR100 with ViT-H-14 at all imbalance ratios above 1, and yields strong performance already at epsilon=0.1 for IR=1.
- Without projection, DPPL-Public reaches 74.0% non-private accuracy on CIFAR100 with ViT-H-14 (vs 72.6% with projection).
- Selecting a public prototype per class takes 34.3 seconds on a single GPU for 100 classes with over one million public images.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use public prototypes selected with DP instead of DP linear probing | CIFAR100 at imbalance ratios greater than 1 · CIFAR100 | balanced accuracy | Helped |
| use DPPL-Mean instead of DPPL-Public | CIFAR100 at low ε · CIFAR100 | balanced accuracy | Hurt |
| use DPPL-Mean instead of DPPL-Public | CIFAR100 at higher ε · CIFAR100 | balanced accuracy | Helped |
| add a projection layer (ablation) | DPPL-Public · CIFAR100, non-private | accuracy: 74.0% -> 72.6% | Hurt |
| add a projection layer (ablation) | DPPL-Mean · CIFAR100 | accuracy | Hurt |
| use multiple Top-K public prototypes per class instead of one (ablation) | DPPL-Public · vision datasets | privacy-utility trade-off | Mixed |
| increase the number of selected prototypes to K=5, K=10, or K=20 (ablation) | DPPL-Public Top-K · CIFAR10, ρ=128 | accuracy: 76.7%, 77.1%, and 77.2% | No clear effect |
| increase the public dataset size (ablation) | DPPL-Public · CIFAR10, CIFAR100, STL10, FOOD101 | accuracy | Helped |
| use ImageNet instead of CC3M as the public prototype pool (ablation) | DPPL-Public · CIFAR100 | accuracy | Helped |
| use a stronger encoder (ablation) | DPPL and comparison methods · CIFAR100 | test accuracy | Helped |
| use the Laplace mechanism instead of the Exponential mechanism for prototype sampling (ablation) | DPPL-Public · CIFAR10, CIFAR100, FOOD101 | accuracy: 91.45 vs 91.45; 36.36 vs 35.89; 19.09 vs 18.82 | No clear effect |
| use the naïve Gaussian mean estimator instead of CoinPress (ablation) | DPPL-Mean · vision datasets | accuracy | Helped |
Surprising findings
- A learned projection layer hurts both DPPL variants even non-privately, indicating misalignment between projection training and public prototype selection.
- DPPL-Mean underperforms at low epsilon because CoinPress-style mean estimation diverges; the naive Gaussian estimator is preferred.
Limitations
- DPPL-Mean degrades at low privacy budgets due to diverging high-dimensional mean estimates.
- Relies on a strong pre-trained encoder; ResNet-50 yields unsatisfactory results for all methods.
Experimental setup
Datasets: CIFAR10 CIFAR100 STL10 FOOD101 ImageNet-1K
Benchmarks: CIFAR10 CIFAR100 STL10 FOOD101
Metrics: balanced accuracy minority class accuracy
Baselines: DP linear probing (DP-SGD) DP-LS DPSGD-Global-Adapt CoinPress
Builds on: prototypical networks DP-SGD exponential mechanism DP-LS DPSGD-Global-Adapt
Closest papers by meaning
- Differential Privacy Under Class Imbalance: Methods and Empirical Insights (ICML 2025 · similarity 0.89)
- Differentially Private Stochastic Gradient Descent with Fixed-Size Minibatches: Tighter RDP Guarantees with or without Replacement (NeurIPS 2024 · similarity 0.89)
- Towards hyperparameter-free optimization with differential privacy (ICLR 2025 · similarity 0.89)
- Optimal Differentially Private Model Training with Public Data (ICML 2024 · similarity 0.89)
- Differentially private optimization for non-decomposable objective functions (ICLR 2025 · similarity 0.88)
- Position: Considerations for Differentially Private Learning with Large-Scale Public Pretraining (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.