Research map / Image generation / Text-to-image diffusion

ORCA: Hunting Compositional Failures in Text-to-Image Diffusion

NeurIPS 2026 · Image generation › Text-to-image diffusion · approach: Compositional text-to-image guidance

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ORCA, an auxiliary training loss that aligns an MMDiT/DiT latent with a frozen low-rank PCA target of DINOv2 features, with the readout subspace chosen by a QR-orthonormalized map conditioned on the T5-minus-CLIP residual, improving FID and GenEval at zero inference cost.

Problem

Text-to-image diffusion models fail on compositional prompts (attribute binding, spatial relations, counting), and it is unclear whether representation alignment with self-supervised visual features helps text-image binding beyond class-conditional generation.

Approach

Align an intermediate diffusion latent with a frozen top-r PCA subspace of DINOv2 features, where the projection subspace is selected per prompt via an orthonormal (QR) basis produced from the residual between a learned linear T5-to-CLIP map and the CLIP embedding; the loss is used only in training.

Key results

• On DiT-L/2 at 200K steps, ORCA reaches FID 16.65 and GenEval 0.291, versus vanilla 400K (FID 24.01, GenEval 0.247) and REG 200K (FID 18.57, GenEval 0.258).
• Gains concentrate on color attribution (about 3x over vanilla at 150K) and position (about 2.9x), per the paper's per-task breakdown.
• Alignment at the last DiT block collapses the model (FID 64.16), while block 8 of 24 is best.

Surprising findings

• Aligning at the final block collapses training (FID 64.16), so the auxiliary loss is only useful at intermediate depth.
• A learnable linear target without regularisation collapses, and the frozen PCA target is needed to avoid it.

Limitations

• Results on MS-COCO at 256x256 only; not validated at web scale.
• Single seed per configuration; bootstrap error bars only, no multi-seed variance.
• Only a single alignment block and a single encoder configuration were tested; encoder-swap study left for future work.

Experimental setup

Datasets: MS-COCO

Benchmarks: MS-COCO 256x256 GenEval

Metrics: FID-30K GenEval CLIPScore PickScore

Baselines: Vanilla diffusion training REPA REG

Builds on: REPA REG MMDiT DiT DINOv2

Closest papers by meaning

• Diffusion Feedback Helps CLIP See Better (ICLR 2025 · similarity 0.90)
• Magnet: We Never Know How Text-to-Image Diffusion Models Work, Until We Learn How Vision-Language Models Function (NeurIPS 2024 · similarity 0.89)
• Progressive Compositionality in Text-to-Image Generative Models (ICLR 2025 · similarity 0.89)
• Diffusion Classifiers Understand Compositionality, but Conditions Apply (NeurIPS 2025 · similarity 0.89)
• CONFORM: Contrast is All You Need for High-Fidelity Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.89)
• I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models (ICML 2025 · similarity 0.89)