Research map / Image generation / Text-to-image diffusion
Universal Few-shot Spatial Control for Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UFC, a few-shot universal spatial control adapter for diffusion models that matches query conditions against support image-condition pairs via patch-wise matching, fine-tuning only small task-specific parameters after episodic meta-training.
Problem
Spatial control adapters for text-to-image diffusion models are trained per task and generalize poorly to novel spatial conditions, requiring thousands of labeled examples for new tasks.
Approach
Represent novel spatial conditions as attention-weighted interpolations of task-agnostic support image patch features, with weights from patch-wise similarity between query and support conditions; meta-train episodically, then fine-tune only bias/LoRA-style task parameters, matching module, and projections on 30 support examples.
Key results
- With 30-shot fine-tuning on unseen tasks, UFC (UNet) achieves Canny SSIM 0.3239 and Normal MAE 15.09, outperforming few-shot baselines including fine-tuned Prompt Diffusion (0.2773) and Uni-ControlNet+FT (0.2222).
- UFC reaches over 90% of fully supervised ControlNet controllability on most conditions, and surpasses fully supervised Uni-ControlNet on several tasks with 150 support examples.
- UFC (DiT) achieves higher controllability than UFC (UNet) on all six tasks, e.g., Pose AP50 0.336 vs 0.229 and Canny SSIM 0.3984 vs 0.3239.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UFC instead of the strongest few-shot baseline | COCO 2017 val, Canny control · Canny | SSIM: 0.2773 -> 0.3239 | Helped |
| Use UFC instead of the strongest few-shot baseline | COCO 2017 val, Pose control · Pose | AP^50: 0.034 -> 0.229 | Helped |
| Use UFC instead of the strongest few-shot baseline | COCO 2017 val, Pose generation · Pose | FID: 43.60 -> 47.91 | Hurt |
| Add patch-wise matching (ablation) | UFC · COCO 2017 val, Canny | SSIM: 0.2984 -> 0.3239 | Helped |
| Add patch-wise matching (ablation) | UFC · COCO 2017 val, HED | SSIM: 0.4972 -> 0.5121 | Helped |
| Add patch-wise matching (ablation) | UFC · COCO 2017 val, Depth | MSE: 96.17 -> 94.38 | Helped |
| Add patch-wise matching (ablation) | UFC · COCO 2017 val, Normal | MAE: 16.06 -> 15.09 | Helped |
| Add patch-wise matching (ablation) | UFC · COCO 2017 val, Pose | AP^50: 0.150 -> 0.229 | Helped |
| Add patch-wise matching (ablation) | UFC · COCO 2017 val, Densepose | mIoU: 0.3995 -> 0.4340 | Helped |
| Enable test-time fine-tuning (ablation) | UFC · COCO 2017 val, Canny | SSIM: 0.2443 -> 0.3239 | Helped |
| Enable test-time fine-tuning (ablation) | UFC · COCO 2017 val, HED | SSIM: 0.3688 -> 0.5121 | Helped |
| Enable test-time fine-tuning (ablation) | UFC · COCO 2017 val, Depth | MSE: 97.92 -> 94.38 | Helped |
Surprising findings
- Zero-shot Prompt Diffusion and Uni-ControlNet with fine-tuning both perform poorly, while full fine-tuning of Prompt Diffusion yields lower FID on Pose despite near-zero AP50 controllability.
Limitations
- Designed for spatial control generation, not tasks preserving condition image appearance such as colorization, deblurring, or inpainting.
- Requires per-task fine-tuning on a small annotated set; does not adapt via in-context learning as LLMs do.
- Performance depends on support set diversity, which is task-dependent; Pose and Densepose are more sensitive.
Experimental setup
Datasets: LAION-400M COCO 2017 ImageNet-R-TI2I iso3d
Benchmarks: COCO 2017 ImageNet-R-TI2I
Metrics: SSIM AP50 MSE MAE mIoU FID
Baselines: ControlNet Uni-ControlNet Ctrl-X Prompt Diffusion FreeControl
Builds on: ControlNet Stable Diffusion Prompt Diffusion Uni-ControlNet bias-tuning
Closest papers by meaning
- Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model (ICLR 2025 · similarity 0.90)
- Local Conditional Controlling for Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.90)
- CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation (ICLR 2025 · similarity 0.90)
- FlexControl: Computation-Aware Conditional Control with Differentiable Router for Text-to-Image Generation (ICML 2025 · similarity 0.89)
- Simplifying Control Mechanism in Text-to-Image Diffusion Models (AAAI 2025 · similarity 0.89)
- Readout Guidance: Learning Control from Diffusion Features (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.