Research map / Image generation / Text-to-image diffusion

A Cat Is A Cat (Not A Dog!): Unraveling Information Mix-ups in Text-to-Image Encoders through Causal Analysis and Embedding Optimization

NeurIPS 2024 · Image generation › Text-to-image diffusion · approach: Inference-time T2I improvement · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes how the causal text encoder of Stable Diffusion causes first-mentioned-object bias and object mixing/missing, proposes a training-free text embedding balance loss (TEBOpt) applied to text embeddings, and an OWL-ViT-based metric for mixture/missing objects.

Problem

Text-to-image diffusion models suffer from information bias toward the first-mentioned object and object mixing/missing when prompts contain multiple objects; the role of the causal text encoder in this failure was previously unexamined.

Approach

Attributes information bias and loss to causal self-attention in the CLIP text encoder, which accumulates earlier tokens into later ones; a training-free loss optimizes text embeddings to push the later object's embedding away from earlier ones while keeping it close to its pure embedding, complemented by existing latent optimization methods.

Key results

• TEBOpt improves information balance in Stable Diffusion 1.4 by 125.42% (info bias 2.65 to 1.39) on the 400-prompt animal set.
• The proposed OWL-ViT-based metric achieves 81% agreement with human assessments on 400 labeled images.
• On T2I-CompBench color set, TEBOpt raises SD3 two-object coexistence from 68.4% to 76.3% and reduces info bias from 2.07 to 1.30.

Measured effects reported in the paper

Surprising findings

• Masking all given-token embeddings reduces mixture rate but increases missing rate, indicating special tokens (<sot>, <eot>, <pad>) carry general prompt information.
• Directly replacing the later object's embedding with its pure embedding balances information but drops two-object coexistence by 5.25%.

Limitations

• Object mixing and missing remain partly governed by the denoising process, which the text-embedding method does not control.
• Evaluated mainly on equally weighted two-object prompts of 17 animals; less important objects in complex prompts are not handled.
• The method is evaluated on a self-constructed animal prompt set rather than the full T2I-CompBench in the main paper.

Experimental setup

Datasets: T2I-CompBench

Benchmarks: T2I-CompBench COCO 2017 val

Metrics: object mixture rate object missing rate info bias 2-object coexistence rate FID human agreement accuracy token embedding similarity cross-attention map distance

Baselines: Stable Diffusion 1.4 Structure Diffusion Attend-and-Excite SynGen SD 1.5 + ELLA SDXL-Turbo SD3

Builds on: Stable Diffusion Attend-and-Excite SynGen Structure Diffusion

Closest papers by meaning

• LightFair: Towards an Efficient Alternative for Fair T2I Diffusion via Debiasing Pre-trained Text Encoders (NeurIPS 2025 · similarity 0.91)
• Magnet: We Never Know How Text-to-Image Diffusion Models Work, Until We Learn How Vision-Language Models Function (NeurIPS 2024 · similarity 0.90)
• Diffusion Adaptive Text Embedding for Text-to-Image Diffusion Models (NeurIPS 2025 · similarity 0.90)
• Get What You Want, Not What You Don't: Image Content Suppression for Text-to-Image Diffusion Models (ICLR 2024 · similarity 0.89)
• ORCA: Hunting Compositional Failures in Text-to-Image Diffusion (NeurIPS 2026 · similarity 0.88)
• CONFORM: Contrast is All You Need for High-Fidelity Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.88)