Research map / Federated learning and privacy / Privacy attacks and secure learning
DECO-Bench: Unified Benchmark for Decoupled Task-Agnostic Synthetic Data Release
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DECO-Bench, a unified benchmark that integrates adversarial attribute-decoupling methods (CE, CE+Mask, metric learning) with LoRA-finetuned Stable Diffusion synthetic image generation and utility/privacy evaluation on FairFace and UTKFace.
Problem
No standard benchmark exists for systematically comparing task-agnostic attribute decoupling methods for privacy-preserving data release, especially when formal guarantees such as differential privacy are infeasible.
Approach
Integrates adversarial decoupling objectives into LoRA finetuning of a latent diffusion model so that generated synthetic face images anonymize a sensitive attribute (race) while retaining non-sensitive attributes, and evaluates with FID, utility/privacy classifiers, and a pretrained-probe privacy protocol under structured data splits.
Key results
- CE decoupling reduces race classifier accuracy on synthetic FairFace data from 36.60% (real baseline) to 41.64% on a G split, with CE+Mask reaching 37.07%, while gender accuracy drops only from 72.94% to 70.47% (CE).
- PT privacy accuracy drops to around 14-19% for CE and CE+Mask versus about 50% for the real baseline, but trained privacy classifiers still recover race at 37-42%, indicating leakage.
- FID between real and synthetic images is 17.77-37.24, showing a distribution gap attributed to limited LoRA capacity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply cross-entropy decoupling | Stable Diffusion v1-5 · FairFace, Setup A (G) | Privacy CLS accuracy (Race): 50.12 -> 41.64 | Helped |
| Apply cross-entropy decoupling with masking | Stable Diffusion v1-5 · FairFace, Setup A (G) | Privacy CLS accuracy (Race): 50.12 -> 37.07 | Helped |
| Apply cross-entropy decoupling | Stable Diffusion v1-5 · UTKFace, Setup A (G) | Privacy CLS accuracy (Race): 49.47 -> 62.63 | Hurt |
| Apply cross-entropy decoupling with masking | Stable Diffusion v1-5 · UTKFace, Setup A (G) | Privacy CLS accuracy (Race): 49.47 -> 63.64 | Hurt |
| Apply cross-entropy decoupling with masking instead of CE | Stable Diffusion v1-5 · FairFace, Setup A (G) | Privacy CLS accuracy (Race): 41.64 -> 37.07 | Helped |
| Add cross-entropy decoupling with masking instead of CE | Stable Diffusion v1-5 · UTKFace, Setup A (G) | Privacy CLS accuracy (Race): 62.63 -> 63.64 | Hurt |
| Apply cross-entropy decoupling | Stable Diffusion v1-5 · FairFace, Setup A (G) | Utility CLS accuracy (Age): 36.60 -> 33.81 | Hurt |
| Apply cross-entropy decoupling | Stable Diffusion v1-5 · UTKFace, Setup A (G) | Utility CLS accuracy (Age): 72.30 -> 41.17 | Hurt |
| Use CE + Mask instead of CE | Stable Diffusion v1-5 · FairFace, Setup A (G) | FID: 37.24 -> 19.11 | Helped |
| Use CE + Mask instead of CE | Stable Diffusion v1-5 · UTKFace, G split | FID: 32.06 -> 17.77 | Helped |
| Increase classifier training data from G to V+G (ablation) | Decoupling evaluation classifiers · FairFace, Setup A | Privacy CLS accuracy (Race): 41.64 -> 46.23; 37.07 -> 38.22; 45.40 -> 37.49 | Mixed |
| Increase classifier training data from G to Tr+V+G (ablation) | Decoupling evaluation classifiers · FairFace, Setup A | Privacy CLS accuracy (Race): 51.03; 48.87; 54.89 | Mixed |
Surprising findings
- Pretrained-probe privacy (PT) looks near random while classifiers trained on generated data still recover race, indicating information leakage through the generation pipeline.
- Adding differential privacy (CE+DP) gave PT privacy accuracy of 26.18%, which the authors note is not a strong privacy gain relative to CE.
Limitations
- Only one sensitive attribute (race) and one use-case (face images) are studied.
- Generated images show a distribution gap to real data (high FID), and information leakage through non-LoRA weights is not eliminated.
- No error bars are reported across random seeds.
Experimental setup
Datasets: FairFace UTKFace
Benchmarks: FairFace UTKFace
Metrics: classification accuracy (race privacy, age and gender utilit pretrained probe accuracy (PT) FID
Baselines: CE decoupling CE + Mask decoupling Metric Learning decoupling Baseline (Real) data
Builds on: Stable Diffusion LoRA cross-entropy adversarial decoupling filter generating network metric learning
Closest papers by meaning
- MaSS: Multi-attribute Selective Suppression for Utility-preserving Data Transformation from an Information-theoretic Perspective (ICML 2024 · similarity 0.86)
- Does Training with Synthetic Data Truly Protect Privacy? (ICLR 2025 · similarity 0.86)
- Privacy Amplification Persists under Unlimited Synthetic Data Release (NeurIPS 2026 · similarity 0.86)
- A Certified Unlearning Approach without Access to Source Data (ICML 2025 · similarity 0.86)
- Leveraging Per-Instance Privacy for Machine Unlearning (ICML 2025 · similarity 0.85)
- The Utility and Complexity of In- and Out-of-Distribution Machine Unlearning (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.