Research map / Image generation / Text-to-image diffusion
X-Adapter: Adding Universal Compatibility of Plugins for Upgraded Diffusion Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes X-Adapter, a frozen-copy adapter with trainable mapping layers that lets plugins (ControlNet, T2I-Adapter, LoRA) built for Stable Diffusion 1.5 work directly on SDXL without retraining the plugins.
Problem
Plugins such as ControlNet, T2I-Adapter and LoRA are trained for a specific base diffusion model and break when a larger upgraded model (e.g., SDXL) is released, requiring costly retraining of each plugin.
Approach
Keep a frozen copy of the base model (SD 1.5) to preserve plugin connectors, and train only mapping layers that remap its decoder features into the decoder of a frozen upgraded model (SDXL) trained with null text prompts; add a two-stage inference strategy (SD1.5 warm-up then SDXL, SDEdit-style) to align initial latents.
Key results
- On ControlNet with SDXL, X-Adapter reports FID 30.95 and CLIP score 0.2632 versus SD 1.5 at FID 33.09 and CLIP 0.2426 (Table 1).
- User study (5 users) ranks X-Adapter + SDXL at 4.46 for ControlNet result quality and 4.38 for LoRA style fidelity, above SDEdit + SDXL and SD 1.5 (Table 2).
- Decoder-only mapping layers give the strongest guidance; addition fusion works best; two-stage inference with T0 = 4/5 T performs best.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use X-Adapter instead of SDEdit for ControlNet compatibility with SDXL | Stable Diffusion 1.5 ControlNet · COCO validation | Condition reconstruction: 0.14 ± 0.10 -> 0.27 ± 0.13 | Helped |
| Use X-Adapter instead of SDEdit for LoRA compatibility with SDXL | Stable Diffusion 1.5 LoRA · LoRA style control | Style similarity: 0.72 -> 0.83 | Helped |
| Insert mapping layers in the decoder only, rather than encoder or both (ablation) | X-Adapter · Control guidance | Guidance capability | Helped |
| Fuse guidance features by addition rather than guidance fusion or SPADE (ablation) | X-Adapter · Condition-guided generation | Condition restoration | Helped |
| Increase the upgraded model's null-text training probability (ablation) | X-Adapter with SDXL · Condition-guided generation | Guidance ability | Helped |
| Use two-stage inference to align the initial latents (ablation) | X-Adapter with SDXL · ControlNet depth | Conditional accuracy | Helped |
Surprising findings
- Setting the upgraded model's prompt to empty during training (100% null) maximizes the guidance strength of X-Adapter.
- Plugins from SD 1.5 can be combined with SDXL LoRAs in the same pipeline (plugin remix).
Limitations
- Fails to maintain identity for personalized plugins such as IP-Adapter, since they act on the text encoder rather than injected features.
- Two-stage inference requires running the base model for part of the timesteps, adding inference cost.
Experimental setup
Datasets: LAION high-resolution subset (300k images) COCO validation set (5,000 images)
Benchmarks: COCO validation set
Metrics: FID CLIP score condition reconstruction score style similarity user preference rank (AHR)
Baselines: Stable Diffusion 1.5 SDEdit + SDXL
Builds on: ControlNet SDEdit Latent Diffusion Models LoRA T2I-Adapter
Closest papers by meaning
- Remix-DiT: Mixing Diffusion Transformers for Multi-Expert Denoising (NeurIPS 2024 · similarity 0.87)
- Stylus: Automatic Adapter Selection for Diffusion Models (NeurIPS 2024 · similarity 0.87)
- LoRA-X: Bridging Foundation Models with Training-Free Cross-Model Adaptation (ICLR 2025 · similarity 0.87)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.87)
- UniCon: Unidirectional Information Flow for Effective Control of Large-Scale Diffusion Models (ICLR 2025 · similarity 0.86)
- T2I-Adapter: Learning Adapters to Dig Out More Controllable Ability for Text-to-Image Diffusion Models (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.