Research map / Image generation / Fast diffusion sampling
Dense2MoE: Restructuring Diffusion Transformer to MoE for Efficient Text-to-Image Generation
Youwei Zheng, Yuxi Ren, Xin Xia, Xuefeng Xiao, Xiaohua Xie
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MoE conversion of FFNs instead of Diff-Pruning at activated expand ratio 1.5 | FLUX.1 [dev] · GenEval | accuracy: 0.4113 -> 0.5728 | Improved |
| Use MoE at ratio 1.5 instead of Diff-Pruning at ratio 2.0 | FLUX.1 [dev] · GenEval | accuracy: 0.4888 -> 0.5728 | Improved |
| Use MoB grouping instead of Lite depth pruning (9 double, 38 single blocks) | FLUX.1 [dev] · GenEval | accuracy: 0.4919 -> 0.5396 | Improved |
| Use MoB grouping instead of Lite depth pruning (9 double, 26 single blocks) | FLUX.1 [dev] · GenEval | accuracy: 0.0926 -> 0.4956 | Improved |
| Use MoB grouping instead of BK-SDM (9 double, 26 single blocks) | FLUX.1 [dev] · GenEval | accuracy: 0.3450 -> 0.4956 | Improved |
| Use larger shared expert (1S12E2A) instead of smaller shared expert (0.5S14E4A), full MoE (ablation) | FLUX.1 [dev] single-stream blocks · GenEval | accuracy: 0.6270 -> 0.6331 | Improved |
| Use larger shared expert (1S12E2A) instead of smaller (0.5S14E4A), shared-only stage (ablation) | FLUX.1 [dev] single-stream blocks · GenEval | accuracy: 0.6001 -> 0.6187 | Improved |
| Add MoE distillation stage on top of shared-only initialization (1S12E2A) (ablation) | FLUX.1 [dev] single-stream blocks · GenEval | accuracy: 0.6187 -> 0.6331 | Improved |
| Add Taylor metric-based expert initialization (ablation) | MoE initialization · GenEval | accuracy: -0.0013 | Improved |
| Add Taylor metric-based expert initialization (ablation) | MoE initialization · ImageReward | IR: -0.159 | Improved |
| Add dense-to-MoE distillation stage (ablation) | MoE · GenEval | accuracy: -0.0057 | Improved |
| Add dense-to-MoE distillation stage (ablation) | MoE · ImageReward | IR: -0.0204 | Improved |
Closest papers by meaning
- EC-DIT: Scaling Diffusion Transformers with Adaptive Expert-Choice Routing (ICLR 2025 · similarity 0.91)
- Layer- and Timestep-Adaptive Differentiable Token Compression Ratios for Efficient Diffusion Transformers (CVPR 2025 · similarity 0.91)
- Scaling Diffusion Transformers Efficiently via $\mu$P (NeurIPS 2025 · similarity 0.90)
- SparseDiT: Token Sparsification for Efficient Diffusion Transformer (NeurIPS 2025 · similarity 0.90)
- SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformer (ICML 2025 · similarity 0.89)
- Würstchen: An Efficient Architecture for Large-Scale Text-to-Image Diffusion Models (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
