Research map / Image generation / Fast diffusion sampling
Multi-Architecture Multi-Expert Diffusion Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Rebuttal for the AAAI 2024 paper MEME, which assigns distinct architectures (convolution-heavy vs. self-attention) to different diffusion time-step intervals to make efficient diffusion models both compact and high-performing.
Problem
Efficient diffusion models suffer performance degradation; the paper argues different denoising time-steps need different operations, so a single architecture is suboptimal.
Approach
Multi-architecture multi-expert diffusion: each expert is a different U-Net-style architecture (convolution- or self-attention-based) assigned to a time-step interval, so each step uses a specialized, cheaper model.
Key results
- Relative to LDM-L, MEME reduces inference memory by 20.9% (all experts loaded) and speeds up inference by 3.3x.
- Deploying one expert at a time gives a 3.3x reduction in memory use, at the cost of extra model load time.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MEME instead of LDM-L | MEME · Image generation | inference memory cost: 20.9% | Helped |
| use MEME instead of LDM-L | MEME · Image generation | inference speed: 3.3x | Helped |
| load one expert at a time | MEME · Inference | memory use: 3.3x reduction | Helped |
Limitations
- Uses more memory than existing small efficient models when all experts are loaded.
- Additional architecture hyperparameters (e.g., expertization probability) add design complexity.
Experimental setup
Datasets: FFHQ ImageNet
Benchmarks: FFHQ ImageNet
Metrics: inference memory inference speed
Baselines: LDM-L eDiff-I PPAP
Builds on: iFormer LDM eDiff-I PPAP
Closest papers by meaning
- Analyzing and Improving the Training Dynamics of Diffusion Models (CVPR 2024 · similarity 0.90)
- Improving Training Efficiency of Diffusion Models via Multi-Stage Framework and Tailored Multi-Decoder Architecture (CVPR 2024 · similarity 0.90)
- Multistep Distillation of Diffusion Models via Moment Matching (NeurIPS 2024 · similarity 0.89)
- Diffusion Model is Effectively Its Own Teacher (CVPR 2025 · similarity 0.89)
- Plug-and-Play Diffusion Distillation (CVPR 2024 · similarity 0.89)
- A Simple Early Exiting Framework for Accelerated Sampling in Diffusion Models (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.