Research map / Image generation / Autoregressive and latent generation
HiCo: Hierarchical Controllable Diffusion Model for Layout-to-image Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HiCo, a multi-branch weight-sharing ControlNet-style layout-to-image diffusion model with a mask-based Fuse Net, and introduces the HiCo-7K benchmark built from GRIT-20M.
Problem
Layout-to-image generation in complex multi-object scenes suffers from object missing, inconsistent lighting, and distortion; existing training-free and training-based methods struggle to control many objects' positions and appearances jointly.
Approach
Decouple each object's bounding box and caption into a separate weight-shared ControlNet-style branch (plus a global background branch), then fuse branch features with a non-parametric mask-based Fuse Net injected into the UNet upsampling stage, keeping the SD backbone frozen.
Key results
- On HiCo-7K, HiCo reaches FID 14.24 and IS 28.31, versus 16.99 FID for InstanceDiff (next best).
- On HiCo-7K spatial metrics, HiCo achieves AP 21.53 and AR 41.21, above InstanceDiff's AP 16.76 and AR 33.92.
- On COCO-3K, HiCo attains FID 20.02 and AP 36.60, marginally best among compared methods in the FID and AP columns, with LayoutDiffuse slightly higher on AP50.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HiCo instead of LayoutDiffuse | layout-to-image generation · COCO-3K | FID: 20.27 -> 20.02 | Helped |
| Use HiCo instead of LayoutDiffuse | layout-to-image generation · COCO-3K | AP50: 59.59 -> 58.10 | Hurt |
| Use HiCo instead of InstanceDiff | layout-to-image generation · HiCo-7K | FID: 16.99 -> 14.24 | Helped |
| Use HiCo instead of InstanceDiff | layout-to-image generation · HiCo-7K | AP: 16.76 -> 21.53 | Helped |
| Train HiCo on natural scenes and evaluate zero-shot instead of using InstanceDiff | COCO-3K · COCO-3K zero-shot | AR: 78.10 -> 68.40 | Hurt |
| Use HiCo instead of InstanceDiff | layout-to-image generation · HiCo-7K human evaluation | Overall human score: 70.54 -> 78.08 | Helped |
| Add UGC to the HiCo-base configuration with GBB and mask FuseNet (ablation) | HiCo-base · HiCo-7K ablation | FID: 15.26 -> 16.85 | Hurt |
| Add UGC to the HiCo-base configuration with GBB and mask FuseNet (ablation) | HiCo-base · HiCo-7K ablation | AR: 39.51 -> 38.19 | Hurt |
| Add UGC to the HiCo-base configuration with GBB and mask FuseNet (ablation) | HiCo-base · HiCo-7K ablation | LocalCLIP Score: 25.22 -> 25.23 | No clear effect |
| Replace mask FuseNet with summation (ablation) | HiCo-base without UGC, with GBB · HiCo-7K ablation | FID: 15.26 -> 20.81 | Hurt |
| Replace mask FuseNet with averaging (ablation) | HiCo-base without UGC, with GBB · HiCo-7K ablation | FID: 15.26 -> 18.78 | Hurt |
| Replace mask FuseNet with averaging (ablation) | HiCo-base without UGC, with GBB · HiCo-7K ablation | LocalIoU Score: 59.07 -> 56.67 | Hurt |
Surprising findings
- On zero-shot COCO-3K evaluation, HiCo's controllability is not best; the authors attribute this to training on long fine-grained captions being out of distribution for COCO.
- Stacking LoRA on the full UNet degrades results versus LoRA on the HiCo branch, and the model still struggles with multi-concept LoRA layouts.
Limitations
- Struggles with complex occlusion order of overlapping objects due to lack of occlusion-order training data.
- Weak handling of complex layouts combining multiple LoRA concepts.
- Zero-shot controllability on COCO-3K is not the best among compared methods.
Experimental setup
Datasets: GRIT-20M COCO-Stuff COCO-3K HiCo-7K
Benchmarks: COCO-3K HiCo-7K
Metrics: FID YOLO score AP AP50 AP75 AR IS LPIPS LocalCLIP Score LocalIoU Score
Baselines: GLIGEN CAG MIGC InstanceDiff MtDM LAMA LDM LayoutDiffuse LayoutDiffusion
Builds on: ControlNet IP-Adapter Stable Diffusion GRIT-20M LoRA
Closest papers by meaning
- Build-A-Scene: Interactive 3D Layout Control for Diffusion-Based Image Generation (ICLR 2025 · similarity 0.90)
- InstanceAssemble: Layout-Aware Image Generation via Instance Assembling Attention (NeurIPS 2025 · similarity 0.88)
- Rethinking The Training And Evaluation of Rich-Context Layout-to-Image Generation (NeurIPS 2024 · similarity 0.87)
- Control and Realism: Best of Both Worlds in Layout-to-Image without Training (ICML 2025 · similarity 0.87)
- OverLayBench: A Benchmark for Layout-to-Image Generation with Dense Overlaps (NeurIPS 2025 · similarity 0.87)
- Visual Layout Composer: Image-Vector Dual Diffusion Model for Design Layout Generation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.