Research map / Image generation / Text-to-image diffusion
Local Conditional Controlling for Text-to-Image Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes training-free local control for text-to-image diffusion, where image conditions (canny, depth, sketch) apply only to a user-defined region while the rest follows the text prompt, using Regional Discriminate Loss, Focused Token Response, and Feature Mask Constraint.
Problem
Existing structure-controlled text-to-image methods (e.g., ControlNet) apply image conditions globally; naively restricting them to a local region causes local control dominance, where the controlled concept dominates and other prompt objects are omitted.
Approach
Operate only at inference: update noised latents with a loss that separates max cross-attention inside vs. outside the control region for non-control tokens, sharpen attention to each token's strongest response, and mask ControlNet features to the local region.
Key results
- On COCO2017, the full method achieves FID 21.86 vs. 27.87 for ControlNet and CLIP Score 0.305 vs. 0.293.
- On Attend-Condition, CLIP T2T improves to 0.804 vs. 0.732 for ControlNet while CLIP Score is 0.285 vs. 0.261.
- Ablation on COCO canny: adding all three components reduces FID to 21.38 from 23.65 baseline, while IOU drops from 0.62 to 0.57.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed local-control method instead of Noise-Mask | Local control with ControlNet · COCO2017 | FID: 22.72 $\pm$ 0.61 -> 21.86 $\pm$ 0.48 | Helped |
| Use the proposed local-control method instead of T2I | Local control with ControlNet · COCO2017 | CLIP Score: 0.295 $\pm$ 0.03 -> 0.305 $\pm$ 0.02 | Helped |
| Use the proposed local-control method instead of Noise-Mask | Local control with ControlNet · COCO2017 | CLIP T2T: 0.789 $\pm$ 0.006 -> 0.801 $\pm$ 0.006 | Helped |
| Use the proposed local-control method instead of ControlNet | Local control with ControlNet · COCO2017 | IOU: 0.55 $\pm$ 0.07 -> 0.51 $\pm$ 0.10 | Hurt |
| Use the proposed local-control method instead of ControlNet | Local control with ControlNet · Attend-Condition | CLIP T2T: 0.784 $\pm$ 0.019 -> 0.804 $\pm$ 0.002 | Helped |
| Use the proposed local-control method instead of ControlNet | Local control with ControlNet · Attend-Condition | IOU: 0.74 $\pm$ 0.04 -> 0.70 $\pm$ 0.05 | Hurt |
| Add RDLoss to the ControlNet baseline (ablation) | Local control, canny · COCO2017 | CLIP T2T: 0.750 -> 0.786 | Helped |
| Add FMC to the ControlNet baseline (ablation) | Local control, canny · COCO2017 | FID: 23.65 -> 21.83 | Helped |
| Add RDLoss and FMC (ablation) | Local control, canny · COCO2017 | CLIP T2T: 0.750 -> 0.802 | Helped |
| Add FTR to RDLoss (ablation) | Local control, canny · COCO2017 | CLIP T2T: 0.786 -> 0.806 | Helped |
| Add FMC to RDLoss and FTR (ablation) | Local control, canny · COCO2017 | FID: 22.90 -> 21.38 | Helped |
| Use Control Concept Matching instead of manually specifying the controlled concept (ablation) | Local control method · Local control examples | control-condition alignment | Helped |
Surprising findings
- Adding RDLoss alone slightly lowers IOU (0.62 to 0.59) and increases FID, and the feature-mask-only variant drops IOU to 0.40, showing a trade-off between text alignment and spatial alignment.
Limitations
- Alignment with the spatial condition (IOU) can decrease when latent updates are applied.
- Attend-Condition dataset is small, so FID is not reported for it.
- Relies on object-token selection via attention, and beta hyperparameter choice (0.8 to 0.9 suggested).
Experimental setup
Datasets: COCO2017 validation set Attend-Condition
Benchmarks: COCO2017 Attend-Condition
Metrics: FID CLIP Score CLIP T2T IOU LPIPS
Baselines: ControlNet T2I-Adapter Noise-Mask Feature-Mask Inpainting
Builds on: Stable Diffusion ControlNet Attend-and-Excite Prompt-to-Prompt
Closest papers by meaning
- Universal Few-shot Spatial Control for Diffusion Models (NeurIPS 2025 · similarity 0.90)
- CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation (ICLR 2025 · similarity 0.89)
- Simplifying Control Mechanism in Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.89)
- Compositional Text-to-Image Synthesis with Attention Map Control of Diffusion Models (AAAI 2024 · similarity 0.89)
- Multi-Region Text-Driven Manipulation of Diffusion Imagery (AAAI 2024 · similarity 0.88)
- Precise Parameter Localization for Textual Generation in Diffusion Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.