Research map / Image generation / Text-to-image diffusion
FlexControl: Computation-Aware Conditional Control with Differentiable Router for Text-to-Image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FlexControl, which copies all diffusion blocks for control and uses a differentiable Gumbel-Sigmoid router with a computation-aware FLOPs loss to dynamically activate control blocks per timestep and sample, evaluated on SD1.5 (UNet) and SD3.0 (DiT).
Problem
ControlNet-style spatial control relies on ad-hoc heuristics for which network blocks to control and which timesteps receive control, with unclear effects on image quality and compute cost.
Approach
Equip every control block with a trainable router that outputs a binary mask (via Gumbel-Sigmoid during training) to skip blocks per timestep and input, trained end-to-end with a FLOPs-targeting cost loss so that control blocks are activated only when useful.
Key results
- On depth-map control with SD1.5, FlexControl(γ=0.5) reaches FID 11.65 vs 17.76 for ControlNet and 12.45 for ControlNet-Large, at roughly half the control-branch FLOPs of ControlNet-Large (280 G vs 561 G).
- Reports a 6.30% RMSE reduction vs ControlNet on depth maps and mIoU gains of 9.87% (canny vs ControlNet) style improvements on segmentation.
- On SD3.0, FlexControl(γ=0.7) reaches FID 20.54, better than ControlNet-Large (21.64) and ControlNet (27.21), at 4.35 T FLOPs vs 6.22 T for ControlNet-Large.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FlexControl dynamic routing instead of ControlNet-Large | SD1.5 controlled generation · Depth Map | FID: 12.45 -> 11.65 | Helped |
| Use FlexControl dynamic routing instead of ControlNet-Large | SD1.5 controlled generation · Canny Edge | FID: 12.92 -> 11.37 | Helped |
| Use FlexControl dynamic routing instead of ControlNet-Large | SD1.5 controlled generation · Segmentation Mask | FID: 16.78 -> 14.80 | Helped |
| Use FlexControl dynamic routing instead of ControlNet-Large | SD1.5 controlled generation · Depth Map | RMSE: 0.2372 -> 0.2358 | Helped |
| Use FlexControl dynamic routing instead of ControlNet-Large | SD1.5 controlled generation · Canny Edge | SSIM: 0.5642 -> 0.5612 | Hurt |
| Use FlexControl dynamic routing instead of ControlNet-Large | SD1.5 controlled generation · Segmentation Mask | mIoU: 0.3668 -> 0.3751 | Helped |
| Reduce control-block budget by setting γ=0.3 rather than γ=0.5 (ablation) | FlexControl · ADE20K segmentation | FID: 14.80 -> 17.21 | Hurt |
| Increase control-block budget from γ=0.5 to γ=0.7 (ablation) | FlexControl · ADE20K segmentation | FID: 14.80 -> 14.71 | Helped |
| Increase control-block budget from γ=0.3 to γ=0.5 (ablation) | FlexControl · LLAVA-558K edge | FID: 24.39 -> 22.47 | Helped |
| Increase control-block budget from γ=0.5 to γ=0.7 (ablation) | FlexControl · LLAVA-558K edge | FID: 22.47 -> 20.54 | Helped |
Surprising findings
- Activation of control blocks is sparse in early denoising steps and dense in late steps, suggesting late steps matter more for controllability.
- Half-activated FlexControl (γ=0.5) outperforms the fully-activated ControlNet-Large on SD1.5 FID, implying brute-force full control is not optimal.
Limitations
- Code is only promised ('will soon be available'), with no released weights or data at submission.
- Gains over ControlNet-Large are marginal or absent on some metrics (e.g., SD3.0 FID at γ=0.3 and 0.5 is worse than ControlNet-Large), and SSIM for SD3.0 is best with ControlNet-Large.
Experimental setup
Datasets: MultiGen-20M LLaVA-558K ADE20K COCO validation
Benchmarks: MultiGen-20M test set COCO validation (canny, segmentation)
Metrics: FID CLIP score RMSE SSIM mIoU FLOPs it/s
Baselines: ControlNet ControlNet++ T2I-Adapter GLIGEN ControlNet-Large VQGAN LDM PIPT
Builds on: ControlNet Stable Diffusion BrushNet Gumbel-Softmax DDIM
Closest papers by meaning
- CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation (ICLR 2025 · similarity 0.91)
- Simplifying Control Mechanism in Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.90)
- Universal Few-shot Spatial Control for Diffusion Models (NeurIPS 2025 · similarity 0.89)
- A Simple Approach to Unifying Diffusion-based Conditional Generation (ICLR 2025 · similarity 0.89)
- FreeControl: Training-Free Spatial Control of Any Text-to-Image Diffusion Model with Any Condition (CVPR 2024 · similarity 0.89)
- EMControl: Adding Conditional Control to Text-to-Image Diffusion Models via Expectation-Maximization (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.