Research map / Federated learning and privacy / Differential privacy
Differentially Private Domain Adaptation with Theoretical Guarantees
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes two (ε,δ)-differentially private supervised domain adaptation algorithms from a public source to a private target, built on a reparameterized weighted-discrepancy objective, with a convex version for linear regression and a non-convex version for Lipschitz-smooth losses.
Problem
Supervised domain adaptation where the target labeled data is private (differential privacy) and the source data is public, aiming for accurate target predictors with privacy and theoretical guarantees.
Approach
Reparameterize sample weights as u_i = 1/q_i so the reweighted adaptation objective (from a generalization bound) becomes jointly convex for linear regression, and smooth via a softmax surrogate for the non-convex case; then apply noisy projected gradient descent with Gaussian noise, with a Laplace-privatized discrepancy estimate.
Key results
- Non-private convex CnvxAdap achieves relative MSE 0.342 on Gas vs 0.381 for DM and 0.441 for KMM (target-only = 1.0).
- Non-private non-convex NCnvxAdap reaches 62.19% on CIFAR-100 vs 58.27% for KMM and 59.51% for target-only training.
- Private CnvxAdap at epsilon=4 is on par with non-private DM on sentiment data, and at epsilon=10 clearly outperforms DM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CnvxAdap∞ instead of KMM | regression domain adaptation · Wind | relative MSE: 1.009 \pm 0.035 -> 0.985 \pm 0.019 | Helped |
| Use CnvxAdap∞ instead of DM | regression domain adaptation · Airline | relative MSE: 1.547 \pm 0.068 -> 0.992 \pm 0.012 | Helped |
| Use CnvxAdap∞ instead of DM | regression domain adaptation · Gas | relative MSE: 0.381 \pm 0.028 -> 0.342 \pm 0.023 | Helped |
| Use CnvxAdap∞ instead of DM | regression domain adaptation · News | relative MSE: 1.006 \pm 0.009 -> 0.995 \pm 0.019 | Helped |
| Use CnvxAdap∞ instead of DM | regression domain adaptation · Slice | relative MSE: 1.218 \pm 0.130 -> 0.992 \pm 0.050 | Helped |
| Use NCnvxAdap∞ instead of training on target data | classification domain adaptation · Adult | accuracy: 91.09 \pm 0.54 -> 91.38 \pm 0.49 | Helped |
| Use NCnvxAdap∞ instead of training on target data | classification domain adaptation · German | accuracy: 76.40 \pm 1.17 -> 77.96 \pm 1.31 | Helped |
| Use NCnvxAdap∞ instead of training on target data | classification domain adaptation · Accent | accuracy: 92.27 \pm 1.10 -> 93.95 \pm 0.78 | Helped |
| Use NCnvxAdap∞ instead of training on target data | classification domain adaptation · CIFAR-100 | accuracy: 59.51 \pm 0.38 -> 62.19 \pm 0.32 | Helped |
| Use NCnvxAdap∞ instead of training on target data | classification domain adaptation · CIFAR-10 | accuracy: 85.18 \pm 0.24 -> 86.44 \pm 0.29 | Helped |
| Use NCnvxAdap∞ instead of training on target data | classification domain adaptation · SVHN | accuracy: 86.28 \pm 0.44 -> 87.34 \pm 0.46 | Helped |
| Use NCnvxAdap∞ instead of training on target data | classification domain adaptation · ImageNet | accuracy: 85.70 \pm 0.29 -> 86.84 \pm 0.38 | Helped |
Surprising findings
- On the sentiment dataset, private CnvxAdap at epsilon=10 outperforms the non-private DM baseline on most source-target pairs.
Limitations
- Private error term scales as sqrt(d log(1/delta))/(epsilon n), so privacy cost is significant for small target samples.
- Non-convex guarantees target stationary points of the gradient mapping, not global minima.
- Smoothness of the non-convex objective requires conditions on m, n, and softmax parameter mu.
Experimental setup
Datasets: Wind Airline Gas News Slice multi-domain sentiment analysis Adult German Accent CIFAR-100 CIFAR-10 SVHN
Benchmarks: UCI regression datasets multi-domain sentiment analysis Adult German Accent CIFAR-100 CIFAR-10 SVHN ImageNet
Metrics: relative MSE accuracy
Baselines: KMM DM noisy minibatch SGD private DM train target only
Builds on: discrepancy minimization generalization bound for weighted adaptation (Cortes et al.) moments accountant Kernel Mean Matching
Closest papers by meaning
- Certification for Differentially Private Prediction in Gradient-Based Training (ICML 2025 · similarity 0.88)
- Oracle-Efficient Differentially Private Learning with Public Data (NeurIPS 2024 · similarity 0.87)
- Position: Considerations for Differentially Private Learning with Large-Scale Public Pretraining (ICML 2024 · similarity 0.86)
- Public-data Assisted Private Stochastic Optimization: Power and Limitations (NeurIPS 2024 · similarity 0.86)
- Online Sensitivity Optimization in Differentially Private Learning (AAAI 2024 · similarity 0.86)
- Differentially private optimization for non-decomposable objective functions (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.