Research map / Image generation / Text-to-image diffusion
CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CtrLoRA, which trains a shared Base ControlNet across nine base conditions with condition-specific LoRAs, then adapts it to new conditions by training only new LoRAs on as few as 1k pairs.
Problem
Training a separate ControlNet for every spatial condition type requires millions of data pairs and hundreds of GPU hours, which is prohibitive for ordinary users wanting new conditions.
Approach
A shared Base ControlNet learns common image-to-image knowledge across multiple conditions, with per-condition LoRAs (on every linear layer) absorbing condition-specific traits; the condition embedding is a pretrained SD VAE instead of a random conv net, speeding convergence and avoiding sudden convergence.
Key results
- On new conditions with 1k training images, CtrLoRA reaches e.g. Lineart LPIPS 0.305 / FID 16.12 versus ControlNet 0.622 / 22.29.
- CtrLoRA uses 37M learnable parameters per new condition (rank 128) versus 361M for ControlNet, a ~90% reduction.
- Base ControlNet on base conditions performs on par with UniControl (e.g., Canny FID 16.65 vs 18.58, but LPIPS 0.388 vs 0.273).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the pretrained VAE instead of the randomly initialized condition embedding network (ablation) | ControlNet adapting to novel conditions · Lineart, Densepose, Inpainting, and Dehazing (1k and 100k images) | LPIPS and FID: Lineart (1k): LPIPS 0.622 -> 0.393; FID 22.29 -> 20.04 | Helped |
| Initialize from the trained Base ControlNet rather than Stable Diffusion ControlNet (ablation) | Novel-condition adaptation · Lineart, Densepose, Inpainting, and Dehazing (1k and 100k images) | LPIPS and FID: Lineart (1k): LPIPS 0.393 -> 0.300; FID 20.04 -> 14.25 | Helped |
| Replace full-parameter training with condition-specific LoRA (ablation) | Base-ControlNet adaptation to novel conditions · Lineart, Densepose, Inpainting, and Dehazing (1k and 100k images) | LPIPS and FID: Lineart (1k): LPIPS 0.300 -> 0.305; FID 14.25 -> 16.12 | Mixed |
| Increase LoRA rank from 32 toward 256 (ablation) | CtrLoRA · LoRA rank ablation | LPIPS | Helped |
| Increase LoRA rank beyond 64 (ablation) | CtrLoRA · LoRA rank ablation | FID | No clear effect |
| Increase training set size from 1k through 50k images (ablation) | CtrLoRA · Novel-condition adaptation | LPIPS and FID | Helped |
| Increase the number of Base ControlNet training conditions from 3 to 6 to 9 (ablation) | CtrLoRA novel-condition adaptation · Lineart, Densepose, Inpainting, and Dehazing | LPIPS and FID: Lineart: LPIPS 0.348 -> 0.324 -> 0.307; FID 15.71 -> 15.59 -> 15.06 | Helped |
Surprising findings
- Replacing the randomly initialized condition embedding with the pretrained VAE removes the sudden convergence phenomenon of ControlNet.
- Color-related conditions such as Palette and color-prompted Lineart converge more slowly than purely spatial conditions.
Limitations
- Color-related conditions (Palette, color Lineart) converge slowly.
- Built on UNet-based Stable Diffusion v1.5; DiT backbones such as SD3 and Flux not yet explored.
- Base training requires about 6000 GPU hours, which is still expensive.
Experimental setup
Datasets: MultiGen-20M COCO2017 HazeWorld Raindrop Danbooru2019
Benchmarks: COCO2017 validation
Metrics: LPIPS FID
Baselines: UniControl ControlNet ControlNet-LITE ControlNet-XS T2I-Adapter ControlLoRA
Builds on: ControlNet LoRA Stable Diffusion UniControl
Closest papers by meaning
- Simplifying Control Mechanism in Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.92)
- FlexControl: Computation-Aware Conditional Control with Differentiable Router for Text-to-Image Generation (ICML 2025 · similarity 0.91)
- Ctrl-X: Controlling Structure and Appearance for Text-To-Image Generation Without Guidance (NeurIPS 2024 · similarity 0.91)
- A Simple Approach to Unifying Diffusion-based Conditional Generation (ICLR 2025 · similarity 0.90)
- Universal Few-shot Spatial Control for Diffusion Models (NeurIPS 2025 · similarity 0.90)
- ControlAR: Controllable Image Generation with Autoregressive Models (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.