Research map / Image generation / Fast diffusion sampling
A Simple Approach to Unifying Diffusion-based Conditional Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniCon, a unified diffusion model that learns a joint distribution over correlated image pairs (image-depth, image-edge, image-pose, identity, appearance) via timestep-disentangled training and joint cross-attention LoRA adapters on Stable Diffusion v1.5, supporting controllable generation, estimation, and joint generation through different sampling schedules.
Problem
Conditional image generation, estimation, and joint generation are typically handled as separate tasks by separate methods; prior unified models rely on multi-stage training, extra architecture, or inpainting masks and many parameters.
Approach
Model the joint distribution of a correlated image pair with a diffusion model using independent (disentangled) noise levels per modality during training, plus parallel joint cross-attention initialized from self-attention and LoRA adapters; different asynchronous noise schedules at inference then realize different conditional tasks without explicit masks.
Key results
- On spatially-aligned conditions, UniCon Depth FID-6K 13.21 and AbsRel 9.26% vs ControlNet 13.68 / 9.85% and Readout-Guidance 18.72 / 23.19%.
- Pose PCK@0.1 of 61.97 for UniCon vs 57.54 for ControlNet and 24.96 for Readout-Guidance.
- Adds about 15% learned parameters to the base model; trained in about 13 hours on two A800 GPUs for 20K steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UniCon instead of ControlNet for depth-conditioned generation | UniCon-Depth · OpenImages depth-conditioned generation | FID-6K: 13.68 -> 13.21 | Helped |
| Use UniCon instead of ControlNet for depth-conditioned generation | UniCon-Depth · OpenImages depth-conditioned generation | AbsRel (%): 9.85 -> 9.26 | Helped |
| Use UniCon instead of ControlNet for soft-edge-conditioned generation | UniCon-SoftEdge · OpenImages soft-edge-conditioned generation | FID-6K: 13.46 -> 12.80 | Helped |
| Use UniCon instead of ControlNet for soft-edge-conditioned generation | UniCon-SoftEdge · OpenImages soft-edge-conditioned generation | EMSE (1e-2): 2.30 -> 2.28 | Helped |
| Use UniCon instead of ControlNet for pose-conditioned generation | UniCon-Pose · OpenImages pose-conditioned generation | FID-6K: 18.61 -> 17.51 | Helped |
| Use UniCon instead of ControlNet for pose-conditioned generation | UniCon-Pose · OpenImages pose-conditioned generation | PCK @ 0.1: 57.54 -> 61.97 | Helped |
| Use UniCon-Depth rather than ZoeDepth for diverse relative depth estimation | UniCon-Depth · DA-2K | Accuracy (%): 89.1 -> 90.5 | Helped |
| Use asynchronous sampling with clean depth input instead of synchronous sampling (ablation) | UniCon-Depth · Depth-to-image generation | FID: 14.78 -> 13.21 | Helped |
| Use asynchronous sampling with clean depth input instead of synchronous sampling (ablation) | UniCon-Depth · Depth-to-image generation | AbsRel: 17.43 -> 9.26 | Helped |
| Use asynchronous sampling with clean depth input instead of synchronous sampling (ablation) | UniCon-Depth · Depth-to-image generation | δ1: 77.00 -> 91.02 | Helped |
| Use asynchronous sampling with clean depth input instead of synchronous sampling (ablation) | UniCon-Depth · Depth-to-image generation | CLIP Score: 32.45 -> 32.11 | Hurt |
| Remove depth loss during training (ablation) | UniCon-Depth · Depth-to-image generation | generation performance | No clear effect |
Surprising findings
- Inference is substantially slower than ControlNet (5.02 vs 10.82 iterations/s), and depth estimation latency (4.0s) is higher than Marigold LCM and other estimators.
- On NYUv2 and ScanNet depth estimation the model still lags behind SOTA diffusion estimators such as Marigold.
Limitations
- Loosely correlated models (ID) show instability and sometimes produce low-quality or misaligned outputs.
- Inpainting performance is less stable, inherited from Stable Diffusion base model.
- Higher inference cost due to unfused LoRA layers; depth estimation lags behind Marigold on NYUv2 and ScanNet.
Experimental setup
Datasets: PascalVOC CelebA Panda-70M OpenImages NYUv2 ScanNet DA-2K
Benchmarks: PascalVOC-based conditional generation on OpenImages (6K images) NYUv2 ScanNet DA-2K
Metrics: FID-6K AbsRel EMSE PCK@0.1 delta1 CLIP Score user preference
Baselines: ControlNet Readout-Guidance JointNet Marigold LooseControl IP-Adapter-Face Uni-ControlNet LDM3D MiDaS DPT ZoeDepth Depth Anything V2
Builds on: Diffusion Forcing ControlNet Stable Diffusion Readout Guidance LoRA
Closest papers by meaning
- CADS: Unleashing the Diversity of Diffusion Models through Condition-Annealed Sampling (ICLR 2024 · similarity 0.91)
- NoiseCtrl: A Sampling-Algorithm-Agnostic Conditional Generation Method for Diffusion Models (CVPR 2025 · similarity 0.91)
- CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation (ICLR 2025 · similarity 0.90)
- CoDi: Conditional Diffusion Distillation for Higher-Fidelity and Faster Image Generation (CVPR 2024 · similarity 0.90)
- DEFT: Efficient Fine-tuning of Diffusion Models by Learning the Generalised $h$-transform (NeurIPS 2024 · similarity 0.90)
- Decoupling Training-Free Guided Diffusion by ADMM (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.