Research map / Image generation / Text-to-image diffusion

One-Way Ticket: Time-Independent Unified Encoder for Distilling Text-to-Image Diffusion Models

CVPR 2025 · Image generation › Text-to-image diffusion · approach: Inference-time T2I improvement · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Distills a SD2.1 text-to-image diffusion model into a one-step, loop-free student using a time-independent unified encoder (encoder run once, decoder reused across K=4 time steps in parallel) plus a KL regularizer on the noise prediction, trained image-free with VSD.

Problem

Distilled text-to-image diffusion models are fast but lose diversity and quality in one-step generation, and the UNet encoder-decoder design repeats redundant computation across denoising steps.

Approach

Share one UNet encoder pass across multiple decoder time steps (time-independent unified encoder) so decoders can run in parallel, and add a KL divergence term that keeps the student's noise prediction close to a standard Gaussian, removing the need for large image datasets.

Key results

• One-step TiUE achieves COCO2014-30K FID 13.09 vs SD-Turbo 19.51 and SwiftBrushv2 15.98 (one-step baselines), with CLIPScore 0.313.
• Removing the KL term raises COCO2014 FID from 13.09 to 14.90 and lowers Density from 0.683 to 0.617.
• TiUE one-step inference takes about 164.7 ms on A100 versus LCM's 236.2 ms and SD-Turbo's 140.0 ms; LCM 4-step is 592.3 ms.

Measured effects reported in the paper

Surprising findings

• Decoders across multiple time steps capture richer semantic information than one-step decoders, which the authors use to motivate sharing a single encoder pass.
• Student trained without any denoising loss still samples reasonably at other time steps (2, 8, 16, 32, 50), though quality drops.

Limitations

• Inference time and memory are slightly higher than SD-Turbo and SwiftBrushv2 in the one-step setting, and the KL/K choices are only partly explored (K=4 chosen without traversing all values).
• Diversity and quality gains are evaluated mainly with FID/CLIPScore and a user study; some metrics (e.g., Precision) trail baselines.

Experimental setup

Datasets: JourneyDB COCO 2014 COCO 2017 AFHQ CelebA-HQ DrawBench PartiPrompts

Benchmarks: COCO 2014-30K COCO 2017-5K AFHQ CelebA-HQ DrawBench PartiPrompts

Metrics: FID CLIPScore Precision Recall F1 Density Coverage inference time (ms) memory (GB)

Baselines: LCM SD-Turbo SwiftBrushv2 SwiftBrush InstaFlow GigaGAN

Builds on: Variational Score Distillation SwiftBrush Faster Diffusion Latent Diffusion Models DDIM

Closest papers by meaning

• Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.93)
• Scaling Down Text Encoders of Text-to-Image Diffusion Models (CVPR 2025 · similarity 0.90)
• Würstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models (ICLR 2024 · similarity 0.89)
• Dual Diffusion for Unified Image Generation and Understanding (CVPR 2025 · similarity 0.89)
• Improving Training Efficiency of Diffusion Models via Multi-Stage Framework and Tailored Multi-Decoder Architecture (CVPR 2024 · similarity 0.89)
• On the Scalability of Diffusion-based Text-to-Image Generation (CVPR 2024 · similarity 0.89)