Research map / Federated learning and privacy / Differential privacy
Optimal Differentially Private Model Training with Public Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Characterizes minimax error of DP model training (mean estimation, ERM, SCO) when public data is available under semi-DP, proving tight bounds and proposing "even more optimal" weighted estimators/SGD that beat naive baselines in constants.
Problem
Optimal worst-case error of differentially private model training when side in-distribution public data is available (semi-DP), and whether public data can beat naive baselines (discard private data or treat public as private) in constants and practice.
Approach
Semi-DP algorithms that add noise only to the private-sample contribution and combine it with raw public samples via tuned weights (Weighted-Gaussian estimator, Semi-Duchi/Semi-PrivUnit local estimator, Semi-DP-SGD), yielding tight minimax rates up to logs and improved constants over throw-away and standard DP baselines.
Key results
- Weighted-Gaussian mean estimator beats both throw-away and standard Gaussian mechanism in MSE; e.g. factor ~1.98 for n=10,000, d=n/100, rho=0.1, n_pub=0.008n.
- On CIFAR-10 the method reduces test error by 8-9% for logistic regression and by at most 18.9% for Wide-ResNet versus naive approaches.
- Local semi-DP mean estimator (Semi-PrivUnit) is optimal including constants; the method converges in a linear regression case where DP-SGD diverges.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the proposed algorithm instead of the naive approaches | CIFAR-10 logistic regression · CIFAR-10 | test error: 8-9% reduction | Helped |
| use the proposed algorithm instead of the naive approaches | CIFAR-10 Wide-ResNet · CIFAR-10 | test error: at most 18.9% reduction | Helped |
| use Weighted-Gaussian instead of throw-away and standard Gaussian estimators | d-dimensional mean estimation · d-dimensional Bernoulli data | MSE: ≈ 1.98 | Helped |
| use the proposed algorithm instead of DP-SGD | linear regression · linear regression (d=50, n=1000, ε=0.01, n_pub/n=0.1) | convergence | Helped |
| use the proposed algorithm instead of PDA-MD | semi-DP model training · synthetic Gaussian linear regression | test loss | Helped |
| remove warm start (ablation) | proposed semi-DP algorithm · synthetic Gaussian linear regression | test loss | No clear effect |
| remove warm start (ablation) | PDA-MD · synthetic Gaussian linear regression | test loss | Hurt |
| increase the public-data fraction (ablation) | proposed semi-DP algorithm · synthetic Gaussian linear regression | test loss | Helped |
| increase the public-data fraction (ablation) | PDA-MD · synthetic Gaussian linear regression | test loss | Mixed |
| clip both private and public gradients (ablation) | proposed semi-DP algorithm · model training experiments | performance | Helped |
| vary the Hessian regularization parameter (ablation) | PDA-MD · complicated tasks | performance | Mixed |
Surprising findings
- For SCO with small epsilon and moderate private data, semi-DP excess empirical risk for ERM can exceed the excess population risk of SCO, contrary to the non-private and standard DP intuition.
- Naive public-data-only throw-away cannot be asymptotically beaten in the worst case; asymptotic improvements over naive approaches are impossible for these problems.
Limitations
- Lower bounds for some regimes are tight only when n_pub = O(n eps/d) or d = O(1)
- Assumes in-distribution public data for mean estimation and SCO; out-of-distribution case left open
- Some lower bound proofs require a symmetry assumption on the algorithm
Experimental setup
Datasets: CIFAR-10 synthetic Gaussian data synthetic Bernoulli data
Benchmarks: CIFAR-10
Metrics: test loss mean squared error test error
Baselines: Throw-away DP-SGD PDA-MD Gaussian mechanism
Builds on: DP-SGD PDA-MD PrivUnit Duchi local randomizer fingerprinting lemma
Closest papers by meaning
- Differentially Private Model Merging (NeurIPS 2026 · similarity 0.90)
- The Limits of Differential Privacy in Online Learning (NeurIPS 2024 · similarity 0.90)
- Public-data Assisted Private Stochastic Optimization: Power and Limitations (NeurIPS 2024 · similarity 0.89)
- It's My Data Too: Private ML for Datasets with Multi-User Training Examples (ICML 2025 · similarity 0.89)
- Noise-Aware Differentially Private Regression via Meta-Learning (NeurIPS 2024 · similarity 0.89)
- Towards hyperparameter-free optimization with differential privacy (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.