Research map / Federated learning and privacy / Differential privacy
Does Training with Synthetic Data Truly Protect Privacy?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Audits four synthetic-data training paradigms (coreset selection, dataset distillation, data-free KD, diffusion-generated data) with worst-case LiRA membership inference on mislabeled canaries and compares them against DPSGD on CIFAR-10, finding that none beats DPSGD on the privacy-utility-efficiency tradeoff.
Problem
Whether training models on synthetic data (without formal differential privacy) actually protects the privacy of the original private training data, given that such methods often implicitly or explicitly claim privacy protection.
Approach
Use worst-case membership inference auditing (LiRA with mislabeled canaries, TPR at 0.1% FPR, many shadow models) on four synthetic-data training paradigms, and compare them fairly against tuned DPSGD baselines at matched accuracy, revealing that low average MIA success can mask visual leakage and that most paradigms leak more than DPSGD.
Key results
- DPSGD (high utility, 90.36% accuracy) achieves 9.31% TPR@0.1% FPR, versus 72.94% for forgetting coreset (92.57% accuracy) and 43.38% for DFKD (92.09% accuracy).
- DM with private initialization reaches 88.89% MI success on initialization canaries (vs 0.16% for non-init), while MTT (OOD) gets 0.18% TPR at 80.42% accuracy yet visual leakage persists for private-initialized MTT.
- Diffusion-based synthetic data (82.64% accuracy) leaks 12.65% TPR@0.1% FPR, more than DPSGD medium (0.40%) but less than DPSGD high (9.31%); training time is 385.4x undefended.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace private-data initialization with noise initialization for DM (ablation) | DM · CIFAR-10 | TPR@0.1% FPR: 18.02% -> 0.24% | Helped |
| Replace private-data initialization with OOD initialization for DM (ablation) | DM · CIFAR-10 | TPR@0.1% FPR: 18.02% -> 0.18% | Helped |
| Replace private-data initialization with OOD initialization for MTT (ablation) | MTT · CIFAR-10 | TPR@0.1% FPR: 1.51% -> 0.18% | Helped |
| Use OOD initialization rather than private initialization (ablation) | MTT · CIFAR-10 | test accuracy: 81.81% -> 80.42% | Hurt |
| Train with coreset selection instead of the undefended baseline | Coreset Selection · CIFAR-10 | TPR@0.1% FPR: 100.00% -> 72.94% | Helped |
| Train using forgetting-selected samples rather than random coreset selection | Coreset selection · CIFAR-10 | TPR@0.1% FPR: 38.70% -> 72.94% | Hurt |
| Use forgetting selection rather than random selection | Coreset selection · CIFAR-10 | test accuracy: 90.56% -> 92.57% | Helped |
| Use dataset distillation with private initialization | DM · CIFAR-10 | TPR@0.1% FPR: 18.02% vs DPSGD (medium) 0.40% | Hurt |
| Use MTT with OOD initialization instead of DPSGD (medium) | MTT (OOD) · CIFAR-10 | test accuracy: 80.42% vs DPSGD (medium) 83.26% | Hurt |
| Use DFKD instead of DPSGD (high) | DFKD · CIFAR-10 | TPR@0.1% FPR: 43.38% vs DPSGD (high) 9.31% | Hurt |
| Use diffusion-generated synthetic data instead of DPSGD (medium) | Diffusion · CIFAR-10 | TPR@0.1% FPR: 12.65% vs DPSGD (medium) 0.40% | Hurt |
| Initialize synthetic data with private samples rather than noise or OOD data (ablation) | DM · CIFAR-10 initialization canaries | membership inference success rate: 88.89% vs 0.16% | Hurt |
Surprising findings
- Low membership inference success for trajectory-matching dataset distillation (MTT) coexists with near-total visual leakage of private data, so MIA alone can give a false sense of privacy.
- Synthetic images from DFKD that are visually dissimilar (SSIM 0.1661) still trigger teacher memorization of mislabeled canaries, with logit correlation near 0.999.
Limitations
- Experiments are restricted to CIFAR-10 with ResNet-18/ConvNet; generalization to other datasets is not shown.
- Mislabeled canaries are used as worst-case proxies; stronger canaries may exist and yield higher leakage.
- Diffusion experiments reduce sampling steps to 10 and use 16 rather than 32 shadow models due to compute cost.
Experimental setup
Datasets: CIFAR-10 CINIC-10
Benchmarks: CIFAR-10
Metrics: TPR@0.1% FPR test accuracy training efficiency (relative training time)
Baselines: DPSGD Forgetting coreset selection Random coreset selection DM DSA MTT DATM DFKD (BN loss) Real-Fake diffusion-based synthetic data
Builds on: LiRA DPSGD Dataset Distillation (DM) MTT Real-Fake
Closest papers by meaning
- A Privacy-Preserving Framework for Generative Model-driven Synthetic Datasets (AAAI 2025 · similarity 0.89)
- Privacy Amplification Through Synthetic Data: Insights from Linear Regression (ICML 2025 · similarity 0.89)
- Privacy Amplification Persists under Unlimited Synthetic Data Release (NeurIPS 2026 · similarity 0.89)
- Differentially Private Sum-Product Networks (ICML 2024 · similarity 0.88)
- Bounding the Excess Risk for Linear Models Trained on Marginal-Preserving, Differentially-Private, Synthetic Data (ICML 2024 · similarity 0.87)
- Position: Considerations for Differentially Private Learning with Large-Scale Public Pretraining (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.