Research map / Federated learning and privacy / Differential privacy

RAPID: Retrieval Augmented Training of Differentially Private Diffusion Models

ICLR 2025 · Federated learning and privacy › Differential privacy · approach: Differentially private synthetic data · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RAPID, which integrates retrieval-augmented generation into DP training of latent diffusion models: public-data trajectories form a knowledge base, and only later denoising steps are trained with DP-SGD on private data.

Problem

Differentially private diffusion models suffer from large utility loss under tight privacy budgets, large memory footprint from very large DP batch sizes, and expensive iterative sampling at inference.

Approach

Use public data to build a knowledge base of diffusion sampling trajectories keyed by contrastively learned features of early latents; during DP fine-tuning, retrieve a similar trajectory for each private sample, skip intermediate steps, and train only the later steps with DP-SGD.

Key results

• On ImageNet32->CIFAR10 with epsilon=1, RAPID reaches FID 63.2 while DpDm fails to produce sensible outputs and DP-LDM gets 79.1.
• On FFHQ32->CelebA32 with epsilon=10, RAPID achieves FID 37.3 (19.1% lower than DP-LDM and 51.8% lower than DpDm) in the unconditional setting.
• RAPID works with batch size 64 by default, whereas DpDm requires batch size 8,192; RAPID's FID is 29.5 at batch size 16 and 26.67 at batch size 256.

Measured effects reported in the paper

Surprising findings

• PrivImage outperforms RAPID on CIFAR10 at epsilon=10 (27.6 vs 25.4 FID is actually RAPID better; the reported exception is CIFAR10 at epsilon=1, where PrivImage achieves 29.8 vs RAPID 63.2)
• Retrieval-augmented training improves existing DP methods (DpDm on CIFAR10 from 82.2 to 54.1 FID), showing the approach is complementary to other DPDM methods.

Limitations

• Relies on access to a public dataset sharing similar high-level layouts with the private data.
• Retrieval uses only the top-1 nearest trajectory, which may limit quality and diversity.
• Compatible only with latent diffusion models because retrieval operates on latents.

Experimental setup

Datasets: EMNIST MNIST ImageNet32 CIFAR10 FFHQ32 FFHQ64 CelebA32 CelebA64 VOC2005

Benchmarks: MNIST CIFAR10 CelebA32 CelebA64

Metrics: FID coverage downstream classifier accuracy top-k retrieval accuracy

Baselines: DpDm DP-LDM DPSDA PrivImage ReDi

Builds on: DP-SGD DDPM DDIM SimCLR Opacus

Closest papers by meaning

• CPR: Retrieval Augmented Generation for Copyright Protection (CVPR 2024 · similarity 0.89)
• Enhancing Privacy-Utility Trade-offs to Mitigate Memorization in Diffusion Models (CVPR 2025 · similarity 0.88)
• PID: Prompt-Independent Data Protection Against Latent Diffusion Models (ICML 2024 · similarity 0.87)
• Differentially Private Synthetic Data via Foundation Model APIs 2: Text (ICML 2024 · similarity 0.87)
• An Efficient Membership Inference Attack for the Diffusion Model by Proximal Initialization (ICLR 2024 · similarity 0.87)
• PCEvolve: Private Contrastive Evolution for Synthetic Dataset Generation via Few-Shot Private Data and Generative APIs (ICML 2025 · similarity 0.87)