Research map / Image generation / Text-to-image diffusion
Correcting Diffusion Generation through Resampling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a particle filtering framework that resamples diffusion denoising trajectories using a discriminator and an object detector to correct missing objects and low image quality in diffusion generation.
Problem
Diffusion models have distributional discrepancies between generated and real images, causing missing object errors in text-to-image generation and low image quality.
Approach
Treat diffusion sampling as sequential Monte Carlo particle filtering, with resampling weights that estimate the likelihood ratio between the real and generated distributions, using a small set of real images for a discriminator and a pre-trained object detector for object mention ratios.
Key results
- PF-Hybrid reaches 75.79% object occurrence on GPT-Synthetic and 68.13% on MS-COCO in human evaluation, above all baselines.
- Outperforms the strongest baseline by 5% in object occurrence and 1.0 FID on MS-COCO.
- Achieves state-of-the-art FID of 1.02 on ImageNet-64 with 4 images and 165 NFE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply PF-HYBRID resampling | Stable Diffusion v2.1-base · GPT-Synthetic | Object occurrence (%) | Helped |
| apply PF-HYBRID resampling | Stable Diffusion v2.1-base · MS-COCO | Object occurrence (%) | Helped |
| add particle filtering to PF-HYBRID (ablation) | PF-HYBRID · GPT-Synthetic | Object occurrence (%): 67.16 -> 72.96 | Helped |
| add particle filtering to PF-HYBRID (ablation) | PF-HYBRID · MS-COCO | FID: 24.18 -> 24.03 | Helped |
| remove the unconditional discriminator and use only the object mention ratio (ablation) | PF-HYBRID · MS-COCO | Object occurrence (%): 83.84 -> 85.79 | Helped |
| remove the unconditional discriminator and use only the object mention ratio (ablation) | PF-HYBRID · MS-COCO | FID: 24.03 -> 25.77 | Hurt |
| add particle filtering to PF-DISCRIMINATOR (ablation) | PF-DISCRIMINATOR · MS-COCO | FID: 23.28 -> 22.91 | Helped |
| add particle filtering to PF-DISCRIMINATOR (ablation) | PF-DISCRIMINATOR · GPT-Synthetic | Object occurrence (%): 57.09 -> 56.86 | No clear effect |
| train the discriminator on object detector output probabilities instead of images (ablation) | PF-DISCRIMINATOR · MS-COCO | Object occurrence (%): 70.96 -> 77.81 | Helped |
| train the discriminator on object detector output probabilities instead of images (ablation) | PF-DISCRIMINATOR · MS-COCO | FID: 22.91 -> 25.31 | Hurt |
Surprising findings
- Discriminator-based resampling tends to ignore missing objects, improving image quality but not object occurrence.
Limitations
- Object-based correction depends on an object detector and only addresses missing objects through that channel.
- Sample selection requires generating multiple images per caption, reducing sample efficiency.
- Discriminator-based correction can overlook missing objects.
Experimental setup
Datasets: MS-COCO GPT-Synthetic FFHQ ImageNet-64
Benchmarks: GPT-Synthetic MS-COCO FFHQ ImageNet-64
Metrics: object occurrence FID human win rate
Baselines: Stable Diffusion v2.1 D-GUIDANCE Spatial-Temporal attention Attend-and-Excite OBJECTSELECT TIFASELECT REWARDSELECT D-SELECT
Builds on: Restart sampler particle filtering for diffusion (Wu et al.) discriminator guidance (D-GUIDANCE) Stable Diffusion
Closest papers by meaning
- Particle Denoising Diffusion Sampler (ICML 2024 · similarity 0.89)
- Improving Diffusion-Based Image Restoration with Error Contraction and Error Correction (AAAI 2024 · similarity 0.88)
- Reducing Spatial Fitting Error in Distillation of Denoising Diffusion Models (AAAI 2024 · similarity 0.88)
- CCS: Controllable and Constrained Sampling with Diffusion Models via Initial Noise Perturbation (NeurIPS 2025 · similarity 0.88)
- One-step Diffusion with Distribution Matching Distillation (CVPR 2024 · similarity 0.88)
- Diffusing Differentiable Representations (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.