Research map / Image generation / Fast diffusion sampling
UniCon: Unidirectional Information Flow for Effective Control of Large-Scale Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes UniCon, an adapter architecture where information flows unidirectionally from a frozen diffusion model to a trainable adapter that alone produces the output, cutting training VRAM and time versus ControlNet.
Problem
Existing control adapters (e.g., ControlNet) backpropagate through the diffusion model, roughly doubling training memory and cost, and they modify intermediate features of a frozen model, limiting control precision and scaling to transformer diffusion models.
Approach
Feed intermediate features of the frozen diffusion model into a trainable adapter (a copy of the full diffusion architecture) that directly outputs the denoised result, so no gradients flow into or are stored for the diffusion model; integrates features via a new zero-initialized connector (ZeroFT) with multiplication and shortcut.
Key results
- UniCon reduces training VRAM by about one-third and increases training speed 2.3x versus ControlNet on DiT while keeping adapter size fixed (abstract).
- On DiT super-resolution, UniCon reaches PSNR 37.34 and FID 20.34 versus ControlNet's 34.82 and 26.43.
- On SD U-Net canny, UniCon achieves SSIM 0.5570 vs ControlNet 0.4895 and T2I-Adapter 0.3936.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use UniCon instead of ControlNet | DiT · Canny | SSIM: 0.4748 -> 0.5458 | Helped |
| Use UniCon instead of ControlNet | DiT · Depth | MSE: 84.65 -> 82.56 | Helped |
| Use UniCon instead of ControlNet | DiT · Pose | mAP: 0.4135 -> 0.4627 | Helped |
| Use UniCon instead of ControlNet | DiT · SR | PSNR: 34.82 -> 37.34 | Helped |
| Use UniCon instead of ControlNet | StableDiffusion-2.1 · Canny | SSIM: 0.4895 -> 0.5570 | Helped |
| Use unidirectional adapter output rather than directly changing diffusion features (ablation) | Full DiT adapter · Canny | SSIM: 0.5053 -> 0.5458 | Helped |
| Use unidirectional adapter output rather than directly changing diffusion features (ablation) | Full DiT adapter · SR | PSNR: 36.53 -> 37.34 | Helped |
| Use unidirectional output with Skip-Layer rather than direct feature modification (ablation) | Skip-Layer DiT adapter · Canny | FID: 49.78 -> 56.93 | Hurt |
| Use unidirectional output with Skip-Layer rather than direct feature modification (ablation) | Skip-Layer DiT adapter · SR | PSNR | Mixed |
| Use ZeroFT rather than ZeroMLP connector (ablation) | DiT UniCon · Canny | SSIM: 0.5343 -> 0.5426 | Helped |
| Use ZeroFT rather than ZeroMLP connector (ablation) | DiT UniCon · SR | FID: 22.99 -> 22.07 | Helped |
| Copy the full model rather than Skip-Layer blocks (ablation) | DiT adapter · SR | PSNR: 35.49 -> 36.53 | Helped |
Surprising findings
- Skip-Layer adapters, which help ControlNet, do not benefit UniCon and the unidirectional design fails to train for SR with Skip-Layer.
- Copying only the encoder (UniCon-Encoder) is ineffective because no adapter remains to process decoder information.
Limitations
- Does not reduce parameter count, so sampling speed improvement is limited versus DiT-ControlNet.
- Skip-Layer copying with unidirectional flow causes training failure/collapse in SR tasks.
Experimental setup
Datasets: LAION
Benchmarks: LAION test subset (1,000 images)
Metrics: SSIM PSNR MSE mAP FID CLIP-IQA MANIQA MUSIQ CLIP-Score LPIPS
Baselines: ControlNet T2I-Adapter ControlNeXt UniControlNet GLIGEN StableSR DiffBIR PASD ShareAttention
Builds on: ControlNet PixArt-alpha Stable Diffusion 2.1 IDDPM LoRA
Closest papers by meaning
- Analyzing and Improving the Training Dynamics of Diffusion Models (CVPR 2024 · similarity 0.88)
- SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse Low-Rank Adaptation (ICLR 2025 · similarity 0.87)
- ALTER: All-in-One Layer Pruning and Temporal Expert Routing for Efficient Diffusion Generation (NeurIPS 2025 · similarity 0.87)
- FreeU: Free Lunch in Diffusion U-Net (CVPR 2024 · similarity 0.87)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.87)
- Plug-and-Play Diffusion Distillation (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.