Research map / Image generation / Text-to-image diffusion
Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Skrr prunes the T5-XXL text encoder of T2I diffusion models by beam-search skipping of MHA/FFN sub-blocks using an MSE discrepancy on conditional and null embeddings, then re-uses adjacent kept sub-blocks to recover quality.
Problem
Text encoders in T2I diffusion models (e.g., T5-XXL) consume up to 8x more memory than denoisers while contributing negligibly to FLOPs, yet compression of them has been largely unexplored; existing LLM blockwise pruning methods are not tailored to T2I.
Approach
Layer pruning for text encoders that scores MHA/FFN sub-blocks by MSE of projected embeddings (conditional and null prompt) through the denoiser's projection layer, searches skip sets with beam search, and re-uses adjacent kept sub-blocks in place of skipped ones, with a bound showing re-use can tighten error versus skip alone.
Key results
- On PixArt-Sigma at ~41.9% sparsity, Skrr reaches GenEval overall 0.442 versus 0.087 for ShortGPT at 40.5% and 0.367 for FinerCut at 41.7%.
- Skrr at 41.9% sparsity reduces pipeline memory from 10.18 GB to 6.46 GB (dense 5.42B params down to 3.43B) on PixArt-Sigma.
- Skrr keeps FID near 19.9-20.2 across sparsity levels 27-42%, close to the dense 22.89 (FID improves relative to dense in some settings).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| prune and re-use text encoder sub-blocks with Skrr instead of LaCo | PixArt-Σ with T5-XXL · GenEval overall, approximately 41% sparsity | GenEval overall: 0.275 -> 0.442 | Helped |
| prune and re-use text encoder sub-blocks with Skrr instead of FinerCut | PixArt-Σ with T5-XXL · GenEval overall, approximately 41% sparsity | GenEval overall: 0.367 -> 0.442 | Helped |
| prune and re-use text encoder sub-blocks with Skrr instead of ShortGPT | PixArt-Σ with T5-XXL · GenEval overall, approximately 41% sparsity | GenEval overall: 0.087 -> 0.442 | Helped |
| prune and re-use text encoder sub-blocks with Skrr instead of FinerCut | PixArt-Σ with T5-XXL · approximately 41% sparsity | FID: 20.36 -> 19.93 | Helped |
| prune and re-use text encoder sub-blocks with Skrr instead of FinerCut | PixArt-Σ with T5-XXL · approximately 41% sparsity | CLIP: 0.308 -> 0.312 | Helped |
| prune and re-use text encoder sub-blocks with Skrr instead of FinerCut | PixArt-Σ with T5-XXL · approximately 41% sparsity | DreamSim: 0.731 -> 0.741 | Helped |
| compress the text encoder with Skrr instead of using the dense encoder | PixArt-Σ pipeline · maximum reported sparsity | Memory: 10.18 -> 6.46 | Helped |
| add Re-use to the Skip-only model (ablation) | Skrr on PixArt-Σ · MS-COCO validation subset, highest sparsity | text-prompt alignment | Helped |
| increase beam size from k=1 to k=2 (ablation) | Skrr on PixArt-Σ · 41.9% sparsity | GenEval Count: 0.372 -> 0.450 | Helped |
| increase beam size from k=2 to k=3 (ablation) | Skrr on PixArt-Σ · 41.9% sparsity | GenEval Count: 0.450 -> 0.450 | No clear effect |
| increase beam size from k=3 to k=4 (ablation) | Skrr on PixArt-Σ · highest-sparsity beam ablation | GenEval Count: 0.450 -> 0.328 | Hurt |
Surprising findings
- Pruning text encoder layers can lower FID while CLIP, DreamSim and GenEval degrade, which the authors attribute to altered null-condition embeddings acting like a guidance method.
- Adding small random noise to the null embedding also reduced FID, supporting the guidance-like explanation.
Limitations
- Evaluated mainly on PixArt-Sigma with GenEval and FID; text alignment (CLIP, GenEval) degrades at high sparsity.
- Relies on a calibration set and beam search whose cost grows with beam size; beam size k=3 chosen empirically.
- Gain over FinerCut at low sparsity is small; FLOPs slightly higher than baselines.
Experimental setup
Datasets: CC12M MS-COCO 30k validation
Benchmarks: MS-COCO 30k GenEval
Metrics: FID CLIP score DreamSim GenEval overall memory (GB) parameters TFLOPs
Baselines: ShortGPT LaCo FinerCut
Builds on: ShortGPT LaCo FinerCut classifier-free guidance
Closest papers by meaning
- Not All Prompts Are Made Equal: Prompt-based Pruning of Text-to-Image Diffusion Models (ICLR 2025 · similarity 0.87)
- Scaling Down Text Encoders of Text-to-Image Diffusion Models (CVPR 2025 · similarity 0.87)
- A Comprehensive Study of Decoder-Only LLMs for Text-to-Image Generation (CVPR 2025 · similarity 0.87)
- One-Way Ticket: Time-Independent Unified Encoder for Distilling Text-to-Image Diffusion Models (CVPR 2025 · similarity 0.87)
- LightFair: Towards an Efficient Alternative for Fair T2I Diffusion via Debiasing Pre-trained Text Encoders (NeurIPS 2025 · similarity 0.87)
- A Cat Is A Cat (Not A Dog!): Unraveling Information Mix-ups in Text-to-Image Encoders through Causal Analysis and Embedding Optimization (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.