Research map / Image generation / Text-to-image diffusion
Shielded Diffusion: Generating Novel and Diverse Images using Sparse Repellency
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPELL, a training-free, sparse repellency guidance that pushes diffusion trajectories away from static protected sets or intra/inter-batch generated images to improve diversity and novelty.
Problem
Text-to-image and class-conditional diffusion models lack diversity across repeated prompts (often producing near-copies) and can regenerate images from protected or training sets.
Approach
Detect when the expected denoised output x0_hat falls inside a radius-r shield around a reference image and apply a minimal, ReLU-weighted correction to the trajectory; the correction is sparse, acting mostly early in sampling, and can use static protected sets or dynamic sets from the same or previous batches.
Key results
- SPELL raises recall from 0.236 to 0.289 and Vendi from 2.527 to 2.695 on Latent Diffusion with FID nearly unchanged (9.501 to 9.554).
- Shielding all 1.2M ImageNet-1k train images with SPELL cuts MDTv2 generated images within the shields from 7.6% to 0.16% (r=60).
- The repellency correction is typically under 5% of the diffusion score magnitude and never exceeds 35%; at t=0.8 only 40% of trajectories have non-zero repellency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SPELL | Latent Diffusion · CC12M text-to-image | Recall: 0.236 $\pm$ 0.003 -> 0.289 $\pm$ 0.003 | Helped |
| add SPELL | SD3-Medium · CC12M text-to-image | Recall: 0.379 $\pm$ 0.004 -> 0.483 $\pm$ 0.002 | Helped |
| add SPELL | Simple Diffusion · CC12M text-to-image | Recall: 0.230 $\pm$ 0.003 -> 0.248 $\pm$ 0.002 | Helped |
| add SPELL | EDMv2 · ImageNet-1k class-to-image | Recall: 0.589 $\pm$ 0.002 -> 0.600 $\pm$ 0.002 | Helped |
| add SPELL | SD3-Medium class-to-image · ImageNet-1k class-to-image | Recall: 0.143 $\pm$ 0.002 -> 0.206 $\pm$ 0.002 | Helped |
| add SPELL | MDTv2 · ImageNet-1k class-to-image | FID: 4.884 $\pm$ 0.052 -> 4.381 $\pm$ 0.047 | Helped |
| add SPELL | SD3-Medium · CC12M text-to-image | FID: 20.103 $\pm$ 0.090 -> 35.174 $\pm$ 0.153 | Hurt |
| add SPELL | EDMv2 · ImageNet-1k train protection | Images within shields: 7.60% -> 0.16% | Helped |
| search more Voronoi cells for protected neighbors (ablation) | SPELL on EDMv2 · ImageNet-1k train protection | Images within shields: 1.08% -> 0.55% -> 0.33% -> 0.22% -> 0.16% | Helped |
| increase the SPELL shield radius (ablation) | Latent Diffusion · CC12M text-to-image | Recall: 0.263 -> 0.287 -> 0.312 -> 0.337 -> 0.359 -> 0.370 | Helped |
| increase the SPELL shield radius (ablation) | Latent Diffusion · CC12M text-to-image | Precision: 0.558 -> 0.552 -> 0.542 -> 0.531 -> 0.518 -> 0.500 | Hurt |
Surprising findings
- On SD3 the Vendi score rises substantially (26% to 37% increases) while coverage decreases, showing coverage can fall as diversity rises when diversity takes different forms.
- SD3 with SPELL shows a large FID increase (20.1 to 35.2) on the CC12M setup, so the FID cost is not always marginal.
Limitations
- Theoretical guarantee holds only when shields are disjoint; overlapping shields can cancel repellency.
- Repellency is applied in VAE latent L2 space, producing visually rather than semantically different outputs.
- Applied to expected x0 under the forward joint, which matches the generative distribution only for the time-reversal SDE, not the probability flow ODE.
Experimental setup
Datasets: CC12M ImageNet-1k
Benchmarks: CC12M soft-label subset ImageNet-1k
Metrics: Recall Vendi score Coverage Precision Density FID FD_DINOv2 CLIP score
Baselines: Particle Guidance Interval Guidance CADS Base model without repellency
Builds on: classifier-free guidance diffusion posterior sampling particle guidance Doob h-transform
Closest papers by meaning
- Diffusion in Diffusion: Cyclic One-Way Diffusion for Text-Vision-Conditioned Generation (ICLR 2024 · similarity 0.89)
- Simplified and Generalized Masked Diffusion for Discrete Data (NeurIPS 2024 · similarity 0.89)
- Plug-and-Play Diffusion Distillation (CVPR 2024 · similarity 0.89)
- Nearly Zero-Cost Protection Against Mimicry by Personalized Diffusion Models (CVPR 2025 · similarity 0.89)
- Anti-Diffusion: Preventing Abuse of Modifications of Diffusion-Based Models (AAAI 2025 · similarity 0.89)
- Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.