Research map / Image generation / Text-to-image diffusion
Upcycling Text-to-Image Diffusion Models for Multi-Task Capabilities
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Multi-Task Upcycling (MTU), which splits the FFN layers of a pre-trained text-to-image diffusion model into smaller experts combined by a task-conditioned router, so one iso-FLOP model handles T2I, image editing, super-resolution and inpainting.
Problem
Adapting text-to-image diffusion models to multiple image-to-image tasks usually requires costly re-training or extra parameters, which hurts on-device deployment.
Approach
Observing that FFN layers deviate most during task fine-tuning, MTU upcycles dense FFNs into fine-grained experts (d_expert = d_FFN/N) with a softmax router driven by learnable task embeddings, plus task-specific input conv and layer norms, training only these parts with the backbone frozen.
Key results
- MTU on SDXL reaches FID 3.9 on T2I at 1.54 TFLOPs, matching the pre-trained model's cost, versus 4.1 for the single-task T2I model.
- MTU (SDv1.5, N=4) scores 17.2 on image editing I-T directional similarity vs 15.4 for single-task full fine-tuning, and LPIPS 24.8 on super-resolution vs 38.0.
- On SDXL, MTU outperforms LoRA and IA3 on FFN layers across tasks, e.g., 20.1 vs 17.3 (full FT) on image editing and 26.5 vs 26.9 on SR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply MTU instead of the single-task T2I baseline | SDv1.5 · COCO Captions test | FID: 12.9 -> 7.2 | Helped |
| apply MTU instead of the single-task image-editing baseline | SDv1.5 · Image Editing | I-T Directional Similarity: 15.4 -> 17.2 | Helped |
| apply MTU instead of the single-task super-resolution baseline | SDv1.5 · Super Resolution | LPIPS: 38.0 -> 24.8 | Helped |
| apply MTU instead of the single-task inpainting baseline | SDv1.5 · Inpainting | I-I Directional Similarity: 46.5 -> 44.0 | Hurt |
| apply MTU instead of the single-task T2I baseline | SDXL · COCO Captions test | FID: 4.1 -> 3.9 | Helped |
| apply MTU instead of the single-task image-editing baseline | SDXL · Image Editing | I-T Directional Similarity: 17.3 -> 20.1 | Helped |
| apply MTU instead of the single-task super-resolution baseline | SDXL · Super Resolution | LPIPS: 26.9 -> 26.5 | Helped |
| apply MTU instead of the single-task inpainting baseline | SDXL · Inpainting | I-I Directional Similarity: 43.2 -> 44.2 | Helped |
| use four experts rather than one (ablation) | MTU on SDv1.5 · Image Editing | I-T Directional Similarity: 17.6 -> 17.2 | No clear effect |
| use two experts rather than four (ablation) | MTU on SDv1.5 · Image Editing | I-T Directional Similarity: 17.2 -> 9.0 | Hurt |
| use eight experts rather than four (ablation) | MTU on SDv1.5 · Image Editing | I-T Directional Similarity: 17.2 -> 10.2 | Hurt |
| use sixteen experts rather than four (ablation) | MTU on SDv1.5 · Image Editing | I-T Directional Similarity: 17.2 -> 9.6 | Hurt |
Surprising findings
- Increasing the number of experts beyond the dense parameter budget with top-k routing performed worse than combining all experts.
- For SDXL a single expert (N=1) works best; splitting FFNs into more experts degraded performance, and freezing FFNs while tuning only layer norms failed to converge.
Limitations
- Multi-task baselines (Versatile Diffusion, UniDiffuser) were trained under different settings and do not cover all tasks, limiting comparison.
- SDv1.5 performance relies on N=4 experts tuned by ablation; more experts degrade results, and the recipe is not uniformly effective across models.
- Only four image-to-image/generation tasks evaluated, with metrics that are directional CLIP similarities and FID rather than human evaluation.
Experimental setup
Datasets: COCO Captions InstructPix2Pix Real-ESRGAN GQA-Inpaint
Benchmarks: COCO Captions test set (FID) InstructPix2Pix test split (I-T directional similarity) Real-ESRGAN-based SR test (LPIPS) GQA-Inpaint test (I-I directional similarity)
Metrics: FID I-T directional similarity LPIPS I-I directional similarity TFLOPs parameters
Baselines: InstructPix2Pix Full fine-tuning single-task SD models Versatile Diffusion UniDiffuser LoRA IA3
Builds on: sparse upcycling fine-grained MoE Stable Diffusion LoRA IA3
Closest papers by meaning
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.88)
- Not All Prompts Are Made Equal: Prompt-based Pruning of Text-to-Image Diffusion Models (ICLR 2025 · similarity 0.88)
- Diff-Plugin: Revitalizing Details for Diffusion-based Low-level Tasks (CVPR 2024 · similarity 0.88)
- OneActor: Consistent Subject Generation via Cluster-Conditioned Guidance (NeurIPS 2024 · similarity 0.87)
- Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs (ICML 2024 · similarity 0.87)
- Towards Understanding Cross and Self-Attention in Stable Diffusion for Text-Guided Image Editing (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.