Research map / Image generation / Text-to-image diffusion
Ctrl-U: Robust Conditional Image Generation via Uncertainty-aware Reward Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Ctrl-U, which estimates reward uncertainty from the variance of reward-model feedback across two diffusion forward passes at different timesteps and uses it to reweight the consistency loss when fine-tuning ControlNet.
Problem
Reward models used to enforce condition alignment in controllable diffusion give inaccurate feedback on newly generated, noisy/diverse images, which corrupts reward-based fine-tuning of conditional generators.
Approach
Treat the discrepancy between reward-model predictions on two generations from the same condition at different timesteps as an uncertainty indicator (KL divergence for segmentation, L1 for edge/depth), and use it to adaptively down-weight the consistency loss via exp(-U) with a regularization term lambda*U, requiring no extra parameters.
Key results
- Ctrl-U reaches 46.49 mIoU on ADE20K vs 43.64 for ControlNet++ and 49.91 on COCO-Stuff vs 34.56 for ControlNet++ (SD1.5).
- Ctrl-U depth RMSE 25.86 vs 28.32 for ControlNet++; Hed SSIM 0.8401 vs 0.8097.
- FID improves to 11.59 on MultiGen-20M Hed edge vs 15.01 for ControlNet++ (SD1.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Ctrl-U uncertainty-aware reward fine-tuning instead of ControlNet++ | ControlNet++ with SD1.5 · ADE20K segmentation | mIoU: 43.64 -> 46.49 | Helped |
| Use Ctrl-U uncertainty-aware reward fine-tuning instead of ControlNet++ | ControlNet++ with SD1.5 · COCO-Stuff segmentation | mIoU: 34.56 -> 49.91 | Helped |
| Use Ctrl-U uncertainty-aware reward fine-tuning instead of ControlNet++ | ControlNet++ with SD1.5 · MultiGen-20M Hed edge | SSIM: 0.8097 -> 0.8401 | Helped |
| Use Ctrl-U uncertainty-aware reward fine-tuning instead of ControlNet++ | ControlNet++ with SD1.5 · MultiGen-20M LineArt edge | SSIM: 0.8399 -> 0.8488 | Helped |
| Use Ctrl-U uncertainty-aware reward fine-tuning instead of ControlNet++ | ControlNet++ with SD1.5 · MultiGen-20M depth | RMSE: 28.32 -> 25.86 | Helped |
| Use Ctrl-U uncertainty-aware reward fine-tuning instead of ControlNet++ | ControlNet++ with SD1.5 · ADE20K, COCO-Stuff, Hed, LineArt, and depth controls | FID: 29.49 -> 28.61; 19.29 -> 15.79; 15.01 -> 11.59; 13.88 -> 11.99; 16.66 -> 15.48 | Helped |
| Set timestep interval |t1-t2| to 1 rather than 0, 3, 5, 7, or 9 (ablation) | Ctrl-U · ADE20K segmentation | FID: 29.01 -> 28.61 (interval 0 to 1); 29.01; 29.81; 29.10; 28.94 | Helped |
| Vary timestep interval |t1-t2| (ablation) | Ctrl-U · ADE20K segmentation | mIoU: 45.33; 46.49; 46.72; 45.94; 45.11; 44.88 | Mixed |
| Set timestep threshold t_thre to 700 rather than 400 (ablation) | Ctrl-U · ADE20K segmentation | mIoU: 46.49 -> 50.11 | Helped |
| Set timestep threshold t_thre to 700 rather than 400 (ablation) | Ctrl-U · ADE20K segmentation | FID: 28.61 -> 34.21 | Hurt |
| Set regularization weight lambda to 1 rather than 0.05 or 0.1 (ablation) | Ctrl-U · ADE20K segmentation | mIoU: 43.06; 43.47 -> 46.49 | Helped |
| Set regularization weight lambda to 1 rather than 0.05 or 0.1 (ablation) | Ctrl-U · ADE20K segmentation | FID: 30.15; 30.60 -> 28.61 | Helped |
Surprising findings
- Reward-model errors persist even at timestep t=0, indicating inaccurate feedback is not only a high-noise issue.
- Directly regressing uncertainty with an auxiliary network tends to overfit (zero or uniform uncertainty), motivating the variance-based estimate.
Limitations
- Relies on two forward passes per sample, and the uncertainty estimate is only reliable at small timesteps (threshold t_thre=400).
- Reward models for edge conditions are ControlNet annotators made differentiable by code modification, and a COCO-Stuff mIoU gain of 44% may reflect evaluation-setup differences; comparisons are only on SD1.5 and a few SDXL results.
Experimental setup
Datasets: ADE20K COCO-Stuff MultiGen-20M
Benchmarks: ADE20K COCO-Stuff MultiGen-20M
Metrics: mIoU SSIM RMSE FID CLIP-score
Baselines: ControlNet++ ControlNet v1.1 T2I-Adapter GLIGEN Uni-ControlNet UniControl
Builds on: ControlNet ControlNet++
Closest papers by meaning
- Focus-N-Fix: Region-Aware Fine-Tuning for Text-to-Image Generation (CVPR 2025 · similarity 0.88)
- Reward-Instruct: A Reward-Centric Approach to Fast Photo-Realistic Image Generation (NeurIPS 2025 · similarity 0.87)
- Confidence-aware Reward Optimization for Fine-tuning Text-to-Image Models (ICLR 2024 · similarity 0.87)
- Towards Understanding and Quantifying Uncertainty for Text-to-Image Generation (CVPR 2025 · similarity 0.87)
- ImagenHub: Standardizing the evaluation of conditional image generation models (ICLR 2024 · similarity 0.87)
- Diff-Instruct with Diffused Reward: Towards Principled One-step Generator RL (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.