Research map / Image generation / Text-to-image diffusion

Enhancing Privacy-Utility Trade-offs to Mitigate Memorization in Diffusion Models

CVPR 2025 · Image generation › Text-to-image diffusion · approach: Inference-time T2I improvement · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PRSS, an inference-time method that modifies classifier-free guidance by re-anchoring the unconditional term to the memorized prompt and searching LLM-generated semantic prompt alternatives, improving the privacy-utility trade-off against memorization in Stable Diffusion.

Problem

Stable Diffusion memorizes training images, creating copyright and privacy risks; existing mitigation methods that enhance privacy substantially reduce text-image alignment (utility).

Approach

Replace the unconditional term in CFG with the memorized-prompt conditional prediction (prompt re-anchoring) so generations are steered away from memorization with less prompt modification, and use LLM-based semantic prompt search in language space to find privacy-safe prompts that stay semantically close to the user prompt.

Key results

• PRSS consistently achieves better privacy-utility trade-offs than prompt engineering baselines for both global and local memorization, with largest gains on global memorization.
• Using the stronger masked detection signal, PRSS reaches state-of-the-art privacy-utility trade-off, surpassing the Ren et al. mitigation.
• Semantic prompt search with GPT-4 (up to 25 alternatives) costs about $0.02 and 0.9 seconds per alternative prompt.

Measured effects reported in the paper

Surprising findings

• Prompt re-anchoring alone yields low utility, while semantic prompt search alone gives limited privacy protection; only their combination gives the best trade-off.

Limitations

• Performance depends on the accuracy of the memorization detection mechanism; failed detection reverts to standard Stable Diffusion.
• Evaluated only on Stable Diffusion v1-4 and a 500-prompt memorization dataset.
• Relies on the SSCD-based memorization metric with a 0.5 threshold.

Experimental setup

Datasets: LAION memorized prompt dataset from Webster et al. (500 prompts, ~300 memorized)

Benchmarks: memorized prompt dataset of 500 prompts (Webster et al.)

Metrics: CLIP score (utility) SSCD score average SSCD 95th percentile percentage of memorized generations localized similarity (LS)

Baselines: prompt engineering (Wen et al.) Ren et al. localized mitigation Somepalli et al. detection/mitigation

Builds on: classifier-free guidance prompt engineering memorization mitigation (Wen et al.) Ren et al. localized memorization mitigation

Closest papers by meaning

• RAPID: Retrieval Augmented Training of Differentially Private Diffusion Models (ICLR 2025 · similarity 0.88)
• Anti-Diffusion: Preventing Abuse of Modifications of Diffusion-Based Models (AAAI 2025 · similarity 0.88)
• PID: Prompt-Independent Data Protection Against Latent Diffusion Models (ICML 2024 · similarity 0.87)
• Detecting, Explaining, and Mitigating Memorization in Diffusion Models (ICLR 2024 · similarity 0.87)
• Six-CD: Benchmarking Concept Removals for Text-to-image Diffusion Models (CVPR 2025 · similarity 0.87)
• Perturb a Model, Not an Image: Towards Robust Privacy Protection via Anti-Personalized Diffusion Models (NeurIPS 2025 · similarity 0.87)