Research map / Image generation / Text-to-image diffusion
Self-correcting LLM-controlled Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SLD, a training-free closed-loop framework where an LLM parser, open-vocabulary detector (OWL-ViT v2), and LLM controller iteratively detect and correct layout errors in generated images via latent-space operations, also usable for image editing.
Problem
Text-to-image diffusion models often fail to follow complex prompts involving numeracy, attribute binding, negation, and spatial relations, and they operate open-loop without checking alignment with the prompt.
Approach
Turns generation into a closed loop: after generating an image, an LLM parses key objects, an open-vocabulary detector localizes them, and an LLM controller proposes corrected bounding boxes; the differences are converted into latent-space add/delete/reposition/attribute-edit operations applied on a base diffusion model (LMD+ or Stable Diffusion), with no additional training.
Key results
- DALL-E 3 with one-round SLD improves average accuracy on LMD benchmark from 52.0% to 78.5% (+26.5).
- LMD+ with one-round SLD improves average accuracy from 79.3% to 88.3% (+9.0).
- On Stable Diffusion, one-round SLD raises average accuracy from 28.5% to 54.5%, and two rounds to 60.0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply 1-round SLD self-correction | DALL-E 3 · LMD benchmark — Average | accuracy: + 26.5 | Helped |
| Apply 1-round SLD self-correction | LMD+ · LMD benchmark — Average | accuracy: + 9.0 | Helped |
| Apply 1-round SLD self-correction | DALL-E 3 · LMD benchmark — Negation | accuracy: 25% -> 90% | Helped |
| Apply 1-round SLD self-correction | LMD+ · LMD benchmark — Numeracy | accuracy: 82% -> 98% | Helped |
| Apply 1-round SLD self-correction | LMD+ · LMD benchmark — Attribute | accuracy: 49% -> 63% | Helped |
| Apply 1-round SLD self-correction | LMD+ · LMD benchmark — Spatial | accuracy: 86% -> 92% | Helped |
| Add one round of SLD self-correction (ablation) | SD · LMD benchmark — Average | accuracy: + 26.0 | Helped |
| Use two rounds of SLD self-correction instead of one (ablation) | SD + 1-round SLD · LMD benchmark — Average | accuracy: 54.5% -> 60.0% | Helped |
| Use two rounds of SLD self-correction instead of one (ablation) | SD + 1-round SLD · LMD benchmark — Negation | accuracy: 69% -> 73% | Helped |
| Use two rounds of SLD self-correction instead of one (ablation) | SD + 1-round SLD · LMD benchmark — Numeracy | accuracy: 55% -> 61% | Helped |
| Use two rounds of SLD self-correction instead of one (ablation) | SD + 1-round SLD · LMD benchmark — Attribute | accuracy: 25% -> 31% | Helped |
| Use two rounds of SLD self-correction instead of one (ablation) | SD + 1-round SLD · LMD benchmark — Spatial | accuracy: 69% -> 75% | Helped |
Surprising findings
- Applying SLD to DALL-E 3 slightly reduces numeracy-related performance? No: DALL-E 3 numeracy rose from 38% to 61%, while attribute binding for LMD+ with SLD remained lower than DALL-E 3 baseline (63% vs 74%).
Limitations
- SAM segmentation can fail on complex-shaped objects (e.g., hair), causing unintended removal.
- Correctness depends on accuracy of the open-vocabulary detector and LLM controller.
- Relies on multiple external models (GPT-4, OWL-ViT v2, SAM), adding pipeline complexity.
Experimental setup
Datasets: LMD benchmark
Benchmarks: LMD benchmark
Metrics: accuracy (open-vocabulary detector-based alignment)
Baselines: MultiDiffusion Backward Guidance BoxDiff LayoutGPT + GLIGEN DALL-E 3 LMD+ InstructPix2Pix DiffEdit
Builds on: LMD DiffEdit SAM OWL-ViT v2
Closest papers by meaning
- Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs (ICML 2024 · similarity 0.87)
- Navigating Text-To-Image Customization: From LyCORIS Fine-Tuning to Model Evaluation (ICLR 2024 · similarity 0.87)
- SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing (CVPR 2024 · similarity 0.86)
- CoDi: Conditional Diffusion Distillation for Higher-Fidelity and Faster Image Generation (CVPR 2024 · similarity 0.86)
- Self-Play Fine-tuning of Diffusion Models for Text-to-image Generation (NeurIPS 2024 · similarity 0.85)
- Diffusion Self-Distillation for Zero-Shot Customized Image Generation (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.