Research map / Image generation / Text-to-image diffusion
T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes T2I-Adapters, small trainable adapter networks that inject spatial structure and color conditions into a frozen Stable Diffusion model, with cubic non-uniform timestep sampling for training.
Problem
Text prompts alone give poor structural and color control over text-to-image diffusion models, and existing control methods (e.g., ControlNet) are heavy or add per-step cost.
Approach
Freeze the pre-trained T2I diffusion model and train lightweight adapters (~77M params) that align external condition maps (sketch, canny, depth, keypoint, segmentation, spatial color) with the model's internal features, injected once into the UNet encoder; adapters are composable by weighted sum and transferable to fine-tuned SD models, with a cubic timestep sampling schedule to force adapters to
Key results
- On COCO validation, T2I-Adapter achieves sketch FID 18.30 versus ControlNet 19.84, and keypoint FID 29.77 versus ControlNet 28.93 (comparable).
- T2I-Adapter uses ~77M parameters versus ControlNet's 567M and runs at 27.96 it/s versus 18.78 it/s on a single A100 with SD-V1.5.
- The 79M T2I-Adapter-XL performs close to the 1250M ControlNet-SDXL on canny control.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use T2I-Adapter instead of ControlNet | SD-V1.5 · COCO validation set, sketch | FID: 19.84 -> 18.30 | Helped |
| Use T2I-Adapter instead of ControlNet | SD-V1.5 · COCO validation set, canny | FID: 15.73 -> 17.96 | Hurt |
| Use T2I-Adapter instead of ControlNet | SD-V1.5 · COCO validation set, depth | FID: 19.09 -> 18.14 | Helped |
| Use T2I-Adapter instead of ControlNet | SD-V1.5 · COCO validation set, keypoint | FID: 28.93 -> 29.77 | Hurt |
| Use T2I-Adapter instead of ControlNet | SD-V1.5 · COCO validation set, semantic segmentation | FID: 18.78 -> 16.78 | Helped |
| Use T2I-Adapter instead of ControlNet | SD-V1.5 · COCO validation set, sketch | CLIP Score: 0.2638 -> 0.2593 | Hurt |
| Use T2I-Adapter instead of ControlNet | SD-V1.5 · COCO validation set, canny | CLIP Score: 0.2613 -> 0.2608 | Hurt |
| Use T2I-Adapter instead of ControlNet | SD-V1.5 · COCO validation set, depth | CLIP Score: 0.2631 -> 0.2656 | Helped |
| Use T2I-Adapter instead of ControlNet | SD-V1.5 · COCO validation set, keypoint | CLIP Score: 0.2640 -> 0.2617 | Hurt |
| Use T2I-Adapter instead of ControlNet | SD-V1.5 · COCO validation set, semantic segmentation | CLIP Score: 0.2653 -> 0.2652 | No clear effect |
| Inject guidance into the UNet encoder instead of the decoder at 4 scales (ablation) | T2I-Adapter · COCO validation set, sketch guidance | MSE: 0.1455 -> 0.1280 | Helped |
| Add encoder guidance to decoder guidance at 4 scales (ablation) | T2I-Adapter · COCO validation set, sketch guidance | MSE: 0.1455 -> 0.1207 | Helped |
Surprising findings
- Adding structure guidance in the middle and late diffusion stages has little effect on output; the main content is determined early in sampling.
- Gains from larger control models (SDXL-sized ControlNet) are limited and do not scale with model size.
Limitations
- Multi-adapter composition requires manual tuning of weights for each condition's strength.
- Color control is coarse, using a spatial palette derived from extreme downsampling rather than precise color specification.
Experimental setup
Datasets: COCO-Stuff LAION-Aesthetics COCO validation set
Benchmarks: COCO validation set
Metrics: FID CLIP score user study MSE of extracted sketch
Baselines: ControlNet GLIGEN PITI SPADE OASIS
Builds on: Stable Diffusion LoRA ControlNet
Closest papers by meaning
- Ranni: Taming Text-to-Image Diffusion for Accurate Instruction Following (CVPR 2024 · similarity 0.91)
- DreamDistribution: Learning Prompt Distribution for Diverse In-distribution Generation (ICLR 2025 · similarity 0.90)
- Image Regeneration: Evaluating Text-to-Image Model via Generating Identical Image with Multimodal Large Language Models (AAAI 2025 · similarity 0.90)
- Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models (CVPR 2024 · similarity 0.89)
- AP-Adapter: Improving Generalization of Automatic Prompts on Unseen Text-to-Image Diffusion Models (NeurIPS 2024 · similarity 0.89)
- SimDA: Simple Diffusion Adapter for Efficient Video Generation (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.