Research map / Image generation / Text-to-image diffusion
RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RealCompo, a training-free framework that combines a text-to-image diffusion model and a spatial-aware (layout/keypoint/segmentation) diffusion model via a cross-attention-driven dynamic balancer that adjusts each model's noise coefficients during denoising.
Problem
Text-to-image diffusion models struggle with multi-object compositional generation, while layout-to-image models improve compositionality but lose realism and aesthetic quality as layout control strengthens.
Approach
A balancer dynamically weights the predicted noise of a T2I model and a spatial-aware diffusion model at each denoising step, updating the weights by gradient descent on a loss that measures how well cross-attention maps align with the layout/region masks; LLM (GPT-4) generates layouts, and a transfer function extends it to keypoints and segmentation maps.
Key results
- On T2I-CompBench, RealCompo reports the best score on all seven sub-tasks, e.g. Color 0.7741 and Numeracy 0.6592, versus PixArt-alpha at 0.6886 and 0.5058 for color and numeracy.
- On realism, RealCompo reaches CLIP score 0.334 and aesthetic score 5.742, above SD v1.4 (0.307, 5.326) and SDXL (0.322, 5.531).
- A user study with 39 users and 1755 votes reports widespread user approval of RealCompo on realism and compositionality.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine SD v1.5 and GLIGEN with the RealCompo balancer instead of using PixArt-$\alpha$ | Text-to-image generation · T2I-CompBench | Color: 0.6886 -> 0.7741 | Helped |
| Combine SD v1.5 and GLIGEN with the RealCompo balancer instead of using PixArt-$\alpha$ | Text-to-image generation · T2I-CompBench | Shape: 0.5582 -> 0.6032 | Helped |
| Combine SD v1.5 and GLIGEN with the RealCompo balancer instead of using PixArt-$\alpha$ | Text-to-image generation · T2I-CompBench | Texture: 0.7044 -> 0.7427 | Helped |
| Combine SD v1.5 and GLIGEN with the RealCompo balancer instead of using GLIGEN | Text-to-image generation · T2I-CompBench | Spatial: 0.2632 -> 0.3173 | Helped |
| Combine SD v1.5 and GLIGEN with the RealCompo balancer instead of using PixArt-$\alpha$ | Text-to-image generation · T2I-CompBench | Non-Spatial: 0.3179 -> 0.3294 | Helped |
| Combine SD v1.5 and GLIGEN with the RealCompo balancer instead of using LMD+ | Text-to-image generation · T2I-CompBench | Numeracy: 0.5762 -> 0.6592 | Helped |
| Combine SD v1.5 and GLIGEN with the RealCompo balancer instead of using PixArt-$\alpha$ | Text-to-image generation · T2I-CompBench | Complex: 0.4117 -> 0.4657 | Helped |
| Combine SD v1.5 and GLIGEN with the RealCompo balancer instead of using TokenCompose v2.1 | Text-to-image generation · COCO validation prompts (3K) | CLIP Score: 0.323 -> 0.334 | Helped |
| Combine SD v1.5 and GLIGEN with the RealCompo balancer instead of using Stable Diffusion XL | Text-to-image generation · COCO validation prompts (3K) | Aesthetic Score: 5.531 -> 5.742 | Helped |
| Add the dynamic balancer to the T2I and layout-to-image noise combination (ablation) | RealCompo · Layout-based compositional generation | Layout alignment | Helped |
Surprising findings
- Restricting layout injection to early denoising steps barely improved realism, since the text-only later steps did not fix the decline in image quality.
Limitations
- Computational cost is higher than a single model because two models are combined and loss gradients are computed.
- Performance depends on the backbone pairing; LayGuide's weaker positioning gave less exact box coverage than GLIGEN.
- Layout generation relies on GPT-4 via in-context learning.
Experimental setup
Datasets: COCO validation set (3K prompts)
Benchmarks: T2I-CompBench COCO validation subset (CLIP score, aesthetic score)
Metrics: T2I-CompBench attribute binding scores spatial non-spatial numeracy complex CLIP score LAION aesthetic score user study votes
Baselines: Stable Diffusion v1.4 Stable Diffusion v2 Structured Diffusion Attend-and-Excite DALL-E 2 Stable Diffusion XL PixArt-alpha GLIGEN LMD+ TokenCompose Layout Guidance InstantStyle
Builds on: GLIGEN ControlNet Attend-and-Excite Layout Guidance LMD
Closest papers by meaning
- TokenCompose: Text-to-Image Diffusion with Token-level Supervision (CVPR 2024 · similarity 0.90)
- Compositional Discrete Latent Code for High Fidelity, Productive Diffusion Models (NeurIPS 2025 · similarity 0.89)
- IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generation (ICLR 2025 · similarity 0.89)
- Any-Size-Diffusion: Toward Efficient Text-Driven Synthesis for Any-Size HD Images (AAAI 2024 · similarity 0.89)
- SwiftBrush: One-Step Text-to-Image Diffusion Model with Variational Score Distillation (CVPR 2024 · similarity 0.88)
- CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.