Research map / Label-efficient and robust learning / Domain adaptation and generalization
Bayesian Power Steering: An Effective Approach for Domain Adaptation of Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Bayesian Power Steering (BPS), a lightweight trainable residual network added to a frozen Stable Diffusion v1.5 UNet that injects task-specific condition features into hidden features, for conditional domain adaptation of diffusion models.
Problem
Adapting large pretrained diffusion models to a small target probability space (task-specific subsets of image distributions) with limited data and conditional control signals.
Approach
Derives via Bayes' theorem and Tweedie's formula that the conditional score equals the pretrained score plus a time-dependent steering term from the additional condition, and realizes this steering term as a feature-space residual network with head-heavy, foot-light structure and per-scale weights, trained with frozen backbone.
Key results
- BPS reaches FID 10.49 (10 epochs) and 10.04 (15 epochs) on COCO17 sketch-to-image vs 18.39 for T2I-Adapter, 19.41 for ControlNet, 20.59 for SD.
- On ADE20K segmentation-to-image, BPS attains mIoU 0.366 vs 0.35 reported for ControlNet.
- BPS CLIP Score is 0.2614, slightly below SD (0.2647) and T2I-Adapter (0.2642).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Bayesian power steering (BPS) adapter added to SD with text+sketch condition | Stable Diffusion V1.5 · COCO17 sketch-to-image | FID: FID 10.49 vs 20.59 for SD, 18.39 for T2I-Adapter, 19.41 for ControlNet | Helped |
| BPS trained longer (10 to 15 epochs) | BPS · COCO17 sketch-to-image | FID: 10.49 to 10.04 | Helped |
| BPS versus ControlNet and SD on CLIP score | BPS · COCO17 sketch-to-image | CLIP Score: 0.2614 vs SD 0.2647 (slightly lower) | Hurt |
| Integrating adapter at all encoder/middle/decoder sites with skip connections (ALL) versus EMD (ablation) | SD with BPS integration · ADE20K segmentation-to-image | mIoU: 0.351 vs 0.351 | No clear effect |
| Removing E-D skip-connection integration (EMD vs ALL) (ablation) | SD with BPS integration · ADE20K segmentation-to-image | mIoU: 0.351 both | No clear effect |
| Integrating at encoder+middle+decoder with time-dependent steering vs. middle+E-D only (ME-D) (ablation) | SD with BPS integration · ADE20K segmentation-to-image | mIoU: ME-D 0.163 vs EMD 0.351 | Hurt |
| Adding encoder-side integration to middle block and decoder (MD vs EMD), i.e., omitting encoder (ablation) | SD with BPS integration · ADE20K segmentation-to-image | mIoU: MD 0.240 vs EMD 0.351 | Hurt |
| Replacing decoder-free encoder+middle (EM) with EMD including decoder (ablation) | SD with BPS integration · ADE20K segmentation-to-image | mIoU: 0.350 vs 0.351 | No clear effect |
| Setting time-step input to a constant (removing time information) (ablation) | BPS · COCO17 sketch-to-image | FID: performance declines throughout training stages | Hurt |
| Decreasing intervention weights w_i (down-weighting encoder and middle block) (ablation) | BPS · Layout-to-image and sketch-to-image | diversity of generated results | Mixed |
| Reducing fine-tuning data down to 40 images (ablation) | BPS · COCO17 sketch-to-image | qualitative sample quality | No clear effect |
| Human preference ranking BPS vs T2I-Adapter and ControlNet | BPS · sketch-to-image user study | User Preference Ranking (1-3): Result Quality 2.38 vs 1.868/1.77; Condition Fidelity 2.52 vs 1.95/1.53 | Helped |
Surprising findings
- ControlNet's CLIP Score (0.2361) is notably lower than the unconditioned SD baseline, despite its control signal.
- Constant time-step input degrades performance, indicating time information matters for steering intensity.
Limitations
- Refined structure needed to capture intervention degree for multiple conditional inputs.
- Intervention weights are manually set and not adaptively adjusted.
- Slightly lower CLIP Score than baselines; text-condition dependence is not modeled (BPS ignores C_text).
Experimental setup
Datasets: COCO17 ADE20K WikiArt synthetic 2D ring data
Benchmarks: COCO17 sketch-to-image COCO layout-to-image ADE20K segmentation-to-image WikiArt artistic drawing
Metrics: FID CLIP Score mIoU user preference ranking classification accuracy (2D)
Baselines: Stable Diffusion T2I-Adapter ControlNet
Builds on: Stable Diffusion ControlNet T2I-Adapter Tweedie's formula
Closest papers by meaning
- A General Framework for Inference-time Scaling and Steering of Diffusion Models (ICML 2025 · similarity 0.87)
- Domain Guidance: A Simple Transfer Approach for a Pre-trained Diffusion Model (ICLR 2025 · similarity 0.87)
- Neural Flow Diffusion Models: Learnable Forward Process for Improved Diffusion Modelling (NeurIPS 2024 · similarity 0.86)
- From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs (NeurIPS 2026 · similarity 0.86)
- Specificity-Aware Diffusion Steering via Variance-Reduced Sequential Monte Carlo (NeurIPS 2026 · similarity 0.86)
- Large Language Models to Diffusion Finetuning (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.