Research map / Image generation / Text-to-image diffusion

Rare Text Semantics Were Always There in Your Diffusion Transformer

NeurIPS 2025 · Image generation › Text-to-image diffusion · approach: Text-driven synthesis and semantic guidance · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ToRA, a training-free intervention for MM-DiTs that combines variance scale-up of text token embeddings (Token Spacing) with rotation of residual PCA space toward the semantic direction (Residual Alignment) to surface rare prompt semantics in text-to-image, text-to-video, and image editing.

Problem

MM-DiT text-to-vision models fail to visually render rare or imaginative prompt concepts that are scarce in pretraining data, even though the semantics seem to exist in text embeddings.

Approach

Scale up the variance of text token embeddings inside the principal PCA subspace (increasing inter-token distances and local isotropy) before each joint-attention block, and rotate the residual subspace toward the CFG semantic vector to avoid side effects, with no training, extra data, or external modules.

Key results

• On RareBench text-to-image, Ours-SD3.0 improves over SD3.0 across all categories (e.g., single-object property GPT4o 89.8 vs 49.4), and R2F+Ours improves over R2F.
• On RareBench text-to-video, Ours-CogVideoX-5B reports gains of over 20 points in most categories (e.g., Shape GPT4o 68.4 vs 50.2 in one category).
• Variance scale-up gives Delta-gamma > 0 in about 75% of random seeds; with Delta-gamma < 0, average CLIP_text was 0.16 vs 0.45 for Delta-gamma > 0 (50 prompts, 100 seeds).

Measured effects reported in the paper

Surprising findings

• Removing top principal components to reduce global anisotropy (a prior NLP remedy) made MM-DiT outputs almost entirely noisy, so global anisotropy was not harmful to semantic emergence.
• Variance scale-up does not always help: in about 25% of seeds it reduces alignment with the semantic vector, motivating residual alignment.

Limitations

• Variance scale-up can amplify embedding directions unrelated to the intended semantics, lowering alignment for some seeds.
• Token Spacing roughly doubles GPU time versus SD3.0 (44.7s vs 21.3s) per the authors' measurement, and R2F is slower still.

Experimental setup

Datasets: RareBench T2I-CompBench GenEval

Benchmarks: RareBench T2I-CompBench GenEval

Metrics: GPT-4o score human evaluation score CLIP score CLIP_img isotropy score GPU time peak memory

Baselines: SD3.0 FLUX-dev FLUX-schnell R2F CogVideoX-2B CogVideoX-5B Stable Flow

Builds on: Diffusion Transformer classifier-free guidance R2F Stable Flow all-but-the-top

Closest papers by meaning

• Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models (NeurIPS 2024 · similarity 0.91)
• Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.91)
• De-Diffusion Makes Text a Strong Cross-Modal Interface (CVPR 2024 · similarity 0.91)
• Exploring the Deep Fusion of Large Language Models and Diffusion Transformers for Text-to-Image Synthesis (CVPR 2025 · similarity 0.91)
• Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis (CVPR 2025 · similarity 0.90)
• Enhancing Text-to-Image Diffusion Transformer via Split-Text Conditioning (NeurIPS 2025 · similarity 0.90)