Research map / Image generation / Fast diffusion sampling
Ensembling Diffusion Models via Adaptive Feature Aggregation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Adaptive Feature Aggregation (AFA), which ensembles multiple fine-tuned Stable Diffusion U-Nets by learning a lightweight spatial-aware block-wise attention aggregator (about 50M trainable parameters) while the base denoisers stay frozen.
Problem
Leveraging multiple high-quality fine-tuned text-to-image diffusion models for stronger generation; existing approaches merge parameters into a static model, ignoring that denoising capabilities vary with prompts, noises, steps, and spatial locations.
Approach
Dynamically aggregate block-wise intermediate features from multiple U-Net denoisers using per-spatial-location attention maps produced by a trainable SABW module conditioned on text and timestep, while keeping all denoisers frozen.
Key results
- On COCO 2017 (Group I), AFA reaches FID 9.76 vs 10.53 for the best baseline (MagicFusion) and 12.32-13.45 for base models.
- On DrawBench, AFA improves ImageReward to 0.4388 (Group I) and 0.4892 (Group II) vs 0.3922 and 0.4396 for the best merging baseline.
- With 20 inference steps AFA keeps ImageReward at 0.4191 (Group I), while base-model-style merging baselines degrade sharply at fewer steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AFA to ensemble Group I models rather than the strongest listed base model | SDv1.5 Group I (ER, MMR, RV) · COCO 2017 | FID: 13.01 -> 9.76 | Helped |
| Use AFA to ensemble Group II models rather than the strongest listed base model | SDv1.5 Group II (AR, CR, RCR) · Draw Bench Prompts | IR: .4238 -> .4892 | Helped |
| Add feature aggregation across U-Net blocks instead of ensembling only the last block (ablation) | AFA · Draw Bench Prompts, Group I | IR: .4176 -> .4388 | Helped |
| Use SABW instead of averaging block output features (ablation) | AFA · Draw Bench Prompts, Group I | IR: .4001 -> .4388 | Helped |
| Use block-wise feature aggregation instead of averaging predicted noises (ablation) | AFA · Draw Bench Prompts, Group I | IR: .3919 -> .4388 | Helped |
| Use spatial attention maps instead of one scalar attention score per model (ablation) | AFA · Draw Bench Prompts, Group I | IR: .4044 -> .4388 | Helped |
| Add timestep conditioning (ablation) | AFA · Draw Bench Prompts, Group I | IR: .4235 -> .4388 | Helped |
| Add textual conditioning (ablation) | AFA · Draw Bench Prompts, Group I | IR: .4163 -> .4388 | Helped |
| Use AFA rather than the original ensembled model at 4 inference steps (ablation) | AFA with LCM distillation · Draw Bench Prompts, Group II | IR: 0.4803 -> -0.5846 | Hurt |
| Increase training samples beyond 10,000 (ablation) | AFA · JourneyDB training | performance | No clear effect |
| Increase the number of ensembled base models (ablation) | AFA · Draw Bench Prompts at 50 inference steps | IR: 0.4238 -> 0.5347 | Helped |
Surprising findings
- Static merging methods (Weighted-Merging, MBW, autoMBW) sometimes underperform the base models on ImageReward, e.g. lower IR in Group I.
- Distilling the ensembled model with LCM while keeping denoisers frozen fails badly (ImageReward around -0.58).
Limitations
- Each inference step costs proportionally more compute since all base models run in parallel, with about 3x per-step time for three models.
- Only applies to ensembling models sharing the same architecture; cannot ensemble heterogeneous architectures.
- Evaluated with a small set of SD1.5 community models; gains on SDXL and FLUX are marginal.
Experimental setup
Datasets: JourneyDB COCO 2017
Benchmarks: COCO 2017 DrawBench prompts
Metrics: FID IS CLIP-I CLIP-T AES PickScore HPSv2 ImageReward
Baselines: Weighted-Merging MBW autoMBW MagicFusion Denoiser-Level MoE Block-Level MoE
Builds on: Stable Diffusion Latent Diffusion Model DDIM MagicFusion
Closest papers by meaning
- APT: Adaptive Personalized Training for Diffusion Models with Limited Data (CVPR 2025 · similarity 0.88)
- Feature Information Dynamics in Diffusion (NeurIPS 2026 · similarity 0.87)
- CleanDIFT: Diffusion Features without Noise (CVPR 2025 · similarity 0.87)
- Temporal Score Analysis for Understanding and Correcting Diffusion Artifacts (CVPR 2025 · similarity 0.87)
- Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference (NeurIPS 2024 · similarity 0.87)
- ALTER: All-in-One Layer Pruning and Temporal Expert Routing for Efficient Diffusion Generation (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.