Research map / Image generation / Fast diffusion sampling
EMControl: Adding Conditional Control to Text-to-Image Diffusion Models via Expectation-Maximization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EMControl, a training-light conditional sampler for text-to-image diffusion that trains a latent forward operator on paired latents and uses EM to estimate a conditional shift and correction at sampling time.
Problem
Adding spatial or other conditional control (edges, depth, segmentation, style) to text-to-image diffusion models efficiently, avoiding the pixel-space mismatch and high training cost of ControlNet-style methods and the latency of training-free pixel-guidance methods.
Approach
Moves guidance from pixel space to latent space by training a latent forward operator A_theta(z_t,t) to map noisy latents to encoded guidance, and treats the condition shift and model-mismatch correction as a blind inverse problem solved by Expectation-Maximization during reverse sampling.
Key results
- On COCO2017 canny condition, EMControl reports CLIP text-image score 0.3035 and CLIP aesthetic 5.4137, with the best aesthetic score across conditions in Table 1.
- Training takes about 20 GPU hours on a single RTX3090, versus 60 to 6900 GPU hours for training-required baselines in Table 2.
- Sampling takes 17 seconds versus 4524 seconds for UniGuid and 115 seconds for FreeDoM under the segmentation setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EMControl instead of ControlNet | conditional text-to-image generation · Depth | FID: 19.3821 -> 25.1024 | Hurt |
| Use EMControl instead of ControlNet | conditional text-to-image generation · Canny | CLIP aesthetic score: 5.2112 -> 5.4137 | Helped |
| Use EMControl instead of ControlNet | conditional text-to-image generation · HED | CLIP aesthetic score: 5.2543 -> 5.4392 | Helped |
| Use EMControl instead of ControlNet | conditional text-to-image generation · Skeleton | CLIP aesthetic score: 5.2403 -> 5.4769 | Helped |
| Use EMControl instead of ControlNet++ | conditional text-to-image generation · HED | FID: 16.2718 -> 24.6031 | Hurt |
| Use EMControl instead of ControlNet++ | conditional text-to-image generation · Canny | Condition Fidelity MSE: 86.7210 -> 70.5733 | Helped |
| Train a latent guidance network and use EM sampling rather than training-required alternatives | conditional text-to-image generation · segmentation | Training (GPU Hours): 500 -> 20 | Helped |
| Use EMControl rather than training-free alternatives | conditional text-to-image generation · segmentation | Sampling (Seconds): 115 -> 17 | Helped |
Surprising findings
- Reported FID on depth (25.10) is worse than ControlNet (19.38) and ControlNet++ (17.41), so EMControl does not lead on FID across all conditions.
Limitations
- Color distortions appear at high guidance scale lambda, as shown in the guidance-scale analysis.
- Quantitative comparison excludes training-free methods due to their long inference times and narrower scope.
- Several baselines lack public models, so some table entries are missing ('-').
Experimental setup
Datasets: COCO2017 WikiArt
Benchmarks: COCO2017 validation set
Metrics: FID CLIP text-image score CLIP aesthetic score SSIM mAP MSE mIoU sampling time training GPU hours
Baselines: ControlNet GLIGEN T2I-Adapter UniCtrl UniCtrlNet ControlNet++ FreeDoM UniGuid DSG
Builds on: ControlNet FreeDoM DDPM Stable Diffusion v1.5
Closest papers by meaning
- FreeControl: Efficient, Training-Free Structural Control via One-Step Attention Extraction (NeurIPS 2025 · similarity 0.89)
- Simplifying Control Mechanism in Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.89)
- FlexControl: Computation-Aware Conditional Control with Differentiable Router for Text-to-Image Generation (ICML 2025 · similarity 0.89)
- Guiding a Diffusion Model with a Bad Version of Itself (NeurIPS 2024 · similarity 0.88)
- NoiseCtrl: A Sampling-Algorithm-Agnostic Conditional Generation Method for Diffusion Models (CVPR 2025 · similarity 0.88)
- Adaptive Guidance: Training-free Acceleration of Conditional Diffusion Models (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.