Research map / Image generation / Text-to-image diffusion
Improving Compositional Generation with Diffusion Models Using Lift Scores
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CompLift, a training-free rejection/resampling criterion that uses lift scores (ELBO difference with and without a condition) from the original diffusion model to improve compositional generation.
Problem
Diffusion models often fail to satisfy complex compositional prompts involving multiple conditions, such as missing objects or incorrect combinations of attributes.
Approach
Approximate lift scores log p(x|c)/p(x) with the denoising-error difference between conditional and unconditional predictions of the same diffusion model, and use the sign to accept or reject samples, composed via logical algebra (product, mixture, negation) without training or external verifiers.
Key results
- On 2D synthetic data, Composable Diffusion + CompLift (T=1000) reaches 99.9% product, 100.0% mixture, and 78.7% negation accuracy versus 56.5%, 70.7%, and 7.9% for Composable Diffusion alone.
- On the Attend-and-Excite benchmark, SD 1.4 + CompLift (T=200) lifts object-animal ImageReward from 0.432 to 1.093, approaching vanilla SD 2.1 (0.924) and exceeding it on that class.
- On CLEVR Position, CompLift keeps accuracy relatively stable as constraints increase, though FID regresses due to reduced diversity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CompLift rejection sampling (T=1000) | Composable Diffusion · 2D synthetic product | accuracy: 56.5 -> 99.9 | Helped |
| Add CompLift rejection sampling (T=1000) | Composable Diffusion · 2D synthetic mixture | accuracy: 70.7 -> 100.0 | Helped |
| Add CompLift rejection sampling (T=1000) | Composable Diffusion · 2D synthetic negation | accuracy: 7.9 -> 78.7 | Helped |
| Add CompLift (T=200) | Stable Diffusion 1.4 · Attend-and-Excite Animals | CLIP: 0.310 -> 0.322 | Helped |
| Add CompLift (T=200) | Stable Diffusion 1.4 · Attend-and-Excite Object&Animal | CLIP: 0.343 -> 0.358 | Helped |
| Add CompLift (T=200) | Stable Diffusion 1.4 · Attend-and-Excite Objects | CLIP: 0.333 -> 0.347 | Helped |
| Add CompLift (T=200) | Stable Diffusion 2.1 · Attend-and-Excite Animals | CLIP: 0.330 -> 0.340 | Helped |
| Add CompLift (T=200) | Stable Diffusion 2.1 · Attend-and-Excite Object&Animal | CLIP: 0.354 -> 0.362 | Helped |
| Add CompLift (T=200) | Stable Diffusion 2.1 · Attend-and-Excite Objects | CLIP: 0.342 -> 0.355 | Helped |
| Add CompLift (T=200) | Stable Diffusion XL · Attend-and-Excite Animals | CLIP: 0.338 -> 0.342 | Helped |
| Add CompLift (T=200) | Stable Diffusion XL · Attend-and-Excite Object&Animal | CLIP: 0.363 -> 0.364 | Helped |
| Add CompLift (T=200) | Stable Diffusion XL · Attend-and-Excite Objects | CLIP: 0.359 -> 0.367 | Helped |
Surprising findings
- Sharing noise across trials introduces bias that can over-reject samples, causing a small regression.
- The composed-prompt score can replace the sampled noise to reduce variance of the lift estimate in text-to-image generation.
Limitations
- Performance depends on the quality of the underlying generative model; poorly trained models may yield no acceptable samples.
- No systematic evaluation of OR/NOT algebras for text-to-image due to lack of benchmarks.
- Rejection reduces sample diversity, causing FID regressions on CLEVR.
Experimental setup
Datasets: 2D synthetic Gaussian mixture and uniform distributions CLEVR Position Attend-and-Excite prompts LAION
Benchmarks: 2D synthetic compositions (Acc, Chamfer Distance) CLEVR Position (accuracy, FID) Attend-and-Excite benchmark (CLIP score, ImageReward)
Metrics: accuracy Chamfer distance FID CLIP score ImageReward
Baselines: Composable Diffusion EBM MCMC (ULA, U-HMC, MALA, HMC) SD 1.4 + EBM (ULA) Attend-and-Excite benchmark models
Builds on: Composable Diffusion Diffusion Classifier Classifier-free guidance Attend-and-Excite
Closest papers by meaning
- IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generation (ICLR 2025 · similarity 0.90)
- Role Bias in Diffusion Models: Diagnosing and Mitigating through Intermediate Decomposition (NeurIPS 2025 · similarity 0.89)
- Composition and Alignment of Diffusion Models using Constrained Learning (NeurIPS 2025 · similarity 0.89)
- CCS: Controllable and Constrained Sampling with Diffusion Models via Initial Noise Perturbation (NeurIPS 2025 · similarity 0.88)
- One-Step Diffusion Distillation through Score Implicit Matching (NeurIPS 2024 · similarity 0.87)
- Diffusion Tree Sampling: Scalable inference‑time alignment of diffusion models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.