Research map / Image generation / Text-to-image diffusion

Semantic Routing: Exploring Multi-Layer LLM Feature Weighting for Diffusion Transformers

NeurIPS 2026 · Image generation › Text-to-image diffusion · approach: Inference-time T2I improvement · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a normalized convex fusion framework with lightweight gates to combine multi-layer LLM hidden states for DiT text conditioning via time-wise, depth-wise, or joint routing, finding depth-wise routing best and time-wise fusion harmed by a train-inference SNR-trajectory mismatch.

Problem

Text conditioning in DiT text-to-image models is static, typically using a single LLM layer across all denoising timesteps and DiT blocks, ignoring the hierarchical semantics of LLM layers and the non-stationary demands of denoising over time and depth.

Approach

Softmax-normalized convex combination of LayerNorm-ed multi-layer LLM hidden states, with fusion logits produced by static, time-conditioned (TCFG MLP on sinusoidal timestep), depth-specific, or joint time-and-depth gates, all trained under a matched backbone for controlled comparison.

Key results

• Depth-wise fusion (S2) reaches 67.07 GenEval and 79.07 GenAI-Bench vs 64.54 and 74.96 for the penultimate-layer baseline (B1).
• On GenAI-Bench Counting, S2 improves +9.97 over B1 and +5.45 over uniform averaging (B2).
• Time-wise fusion (S1) reaches 63.41 GenEval, below the B2 uniform baseline's 66.51, and shows blurriness; manual timestep shift recovers only to 63.65 GenEval.

Surprising findings

• Purely time-wise fusion degrades generation relative to the penultimate-layer baseline on GenEval, attributed to a train-inference SNR trajectory mismatch under CFG.
• The FuseDiT deep-fusion baseline underperforms simple multi-layer averaging despite its lower FLOPs.

Limitations

• Experiments use a single 2.24B DiT trained at 256x256 resolution on a 30M-pair subset.
• Time-wise fusion problem is diagnosed but no robust trajectory-aware fix is provided; the manual shift is heuristic and only partially recovers performance.

Experimental setup

Datasets: LAION-400M subset Qwen3-VL-32B synthetic captions

Benchmarks: GenEval GenAI-Bench DrawBench (UnifiedReward style score)

Metrics: GenEval score GenAI-Bench accuracy UnifiedReward-2.0 style score latent MSE PSNR Params (M) FLOPs (T) latency (ms)

Baselines: Penultimate-layer conditioning Uniform normalized averaging Static learnable fusion FuseDiT

Builds on: FuseDiT SD3 VAE flow matching classifier-free guidance LiDiT

Closest papers by meaning

• Rare Text Semantics Were Always There in Your Diffusion Transformer (NeurIPS 2025 · similarity 0.90)
• Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models (NeurIPS 2024 · similarity 0.89)
• A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation (CVPR 2025 · similarity 0.88)
• Noise Diffusion for Enhancing Semantic Faithfulness in Text-to-Image Synthesis (CVPR 2025 · similarity 0.88)
• Enhancing Text-to-Image Diffusion Transformer via Split-Text Conditioning (NeurIPS 2025 · similarity 0.88)
• Seg4Diff: Unveiling Open-Vocabulary Semantic Segmentation in Text-to-Image Diffusion Transformers (NeurIPS 2025 · similarity 0.87)