Research map / Image generation / Text-to-image diffusion

Compositional Inversion for Stable Diffusion Models

AAAI 2024 · Image generation › Text-to-image diffusion · approach: Multi-concept composition · 11 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes compositional inversion, a post-training add-on to Textual Inversion, DreamBooth, and Custom Diffusion that adds a semantic anchor-attraction loss and a spatial attention-regularization loss so inverted concepts compose with other concepts without dominating the image.

Problem

Inversion methods such as Textual Inversion overfit: inverted concepts dominate generation and other requested concepts go missing when composed in prompts, because reconstruction loss also encodes irrelevant background semantics, pushing embeddings out of distribution.

Approach

Semantic inversion adds a sparse anchor-attraction loss pulling the pseudo-embedding toward embeddings of general COCO-style concepts (the core distribution), and spatial inversion uses an MLP-predicted bounding box per token to add an attention-mask loss that limits dominance of inverted tokens in cross-attention.

Key results

• On pretrained-concept composition, Textual Inversion + semantic + spatial inversion raises CoI Likelihood from 0.032 to 0.284 and Text-Align from 0.603 to 0.702.
• DreamBooth + spatial inversion reaches CoI Likelihood 0.534 vs 0.431 for DreamBooth alone, with Image-Align dropping to 0.657 from 0.734.
• User study: 81.9% of 1,600 sampled images rated above Satisfactory.

Measured effects reported in the paper

Surprising findings

• Augmented Textual Inversion achieves comparable performance to the original Custom Diffusion and DreamBooth without fine-tuning network parameters.
• Semantic inversion occasionally generates low-probability scenes (e.g., half a cat in a backpack), while spatial inversion favors more common scene statistics.

Limitations

• Trade-off: Image-Align decreases by roughly 6.6% to 14.8% relative to the baselines when composing with pretrained concepts.
• Non-rigid objects such as animals are harder to compose, per the user study.
• Spatial inversion relies on an MLP trained on bounding boxes from an object detector over a noun vocabulary.

Experimental setup

Datasets: COCO LAION

Benchmarks: COCO-based composition prompts (1600 prompts, 16000 images)

Metrics: Text-alignment (CLIP similarity) CoI Likelihood (DETR detection) Image-alignment (CLIP image similarity) user study ratings

Baselines: Textual Inversion Custom Diffusion DreamBooth

Builds on: Textual Inversion DreamBooth Custom Diffusion Prompt-to-prompt

Closest papers by meaning

• Improving Compositional Generation with Diffusion Models Using Lift Scores (ICML 2025 · similarity 0.87)
• Inversion-Free Image Editing with Language-Guided Diffusion Models (CVPR 2024 · similarity 0.87)
• Prompt-tuning Latent Diffusion Models for Inverse Problems (ICML 2024 · similarity 0.86)
• Compositional Discrete Latent Code for High Fidelity, Productive Diffusion Models (NeurIPS 2025 · similarity 0.86)
• Semantic Image Inversion and Editing using Rectified Stochastic Differential Equations (ICLR 2025 · similarity 0.86)
• IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generation (ICLR 2025 · similarity 0.86)