Research map / Image generation / Text-to-image diffusion

Scaling Down Text Encoders of Text-to-Image Diffusion Models

CVPR 2025 · Image generation › Text-to-image diffusion · approach: Text-driven synthesis and semantic guidance · 4 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Distills T5-XXL into smaller T5 encoders (down to T5-Base, ~50x smaller) using vision-based knowledge distillation through a frozen Flux.1-dev denoiser, showing text-rendering degrades most with size.

Problem

Large T5-XXL text encoders in diffusion models greatly increase parameter count and GPU memory, and T5 is trained on largely non-visual C4 text, raising the question of whether such a large encoder is needed for text-to-image generation.

Approach

Instead of distilling encoder outputs directly (which causes mode collapse), the student text encoder is trained so that the diffusion model's denoising predictions under student embeddings match those under teacher embeddings, following the teacher's sampling trajectory with a step-following scheme that needs only prompts, not images.

Key results

• T5-Base (220M) distilled from T5-XXL reaches FID 24.32 and CLIP 29.79 on MSCOCO-2014, close to the teacher's 22.36 and 31.30.
• On CommonText sentence-level OCR accuracy, T5-XL scores 48.8 versus 51.2 for T5-XXL, while T5-Small drops to 9.3.
• Replacing T5-XXL with T5-Base cuts Flux pipeline memory from 34113MB to 23401MB with offload and gives over 2.7x latency speedup without offload.

Measured effects reported in the paper

Surprising findings

• Naive output-level distillation of T5 leads to mode collapse in the student's embedding space, while vision-based distillation works better despite non-overlapping embedding distributions.
• Text-rendering collapses below a certain encoder size, while image quality and semantic understanding remain largely intact.

Limitations

• Evaluation of text-rendering relies on a self-constructed template-based CommonText dataset.
• Attribute binding gains depend on training data size, and some metrics are only partially compared against SOTA models.
• Embedding distributions of the distilled encoder no longer overlap with the teacher's, so the mechanism is only partly understood.

Experimental setup

Datasets: LAION-Aesthetics-6.5+ T2I-CompBench CommonText (constructed) MSCOCO-2014 DiffusionDB

Benchmarks: MSCOCO-2014 T2I-CompBench CommonText validation

Metrics: FID CLIP score T2I-CompBench attribute binding, spatial and numeracy scores OCR character/word/sentence accuracy pipeline latency and memory

Baselines: Pixart-Alpha SD3 Flux w/ T5-XXL naive embedding distillation

Builds on: knowledge distillation Flux T5 classifier-free guidance

Closest papers by meaning

• Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models (NeurIPS 2024 · similarity 0.91)
• One-Way Ticket: Time-Independent Unified Encoder for Distilling Text-to-Image Diffusion Models (CVPR 2025 · similarity 0.90)
• A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation (CVPR 2025 · similarity 0.90)
• Würstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models (ICLR 2024 · similarity 0.89)
• Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.89)
• TextCraftor: Your Text Encoder Can be Image Quality Controller (CVPR 2024 · similarity 0.89)