Research map / Architectures and efficiency / Mixture of experts
Breaking the Uniformity Trap: Scaling Video Diffusion Model via SplitMoE
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SplitMoE, a video diffusion MoE that splits experts into prototype-guided semantic and generic pools, upcycled from Wan2.2 low-noise, and reports gains over dense fine-tuning and standard MoE under matched activated parameters.
Problem
Standard token-wise MoE with load-balancing losses forces uniform expert usage, which fragments spatiotemporally coherent regions in video diffusion transformers.
Approach
Split experts into disjoint semantic and generic pools with independent top-K routing per pool; a semantic router is aligned by KL to soft prototype assignments computed from clean VAE features, with pull-push prototype regularization, while the generic pool uses loss-free balancing.
Key results
- Full SplitMoE achieves 58.46% Creativity and 84.47% Human Fidelity on VBench-2, best among listed models on those dimensions, and 84.63% Consistent Attribute on T2V-CompBench.
- Full SplitMoE outperforms Dense Wan2.2-FT and Wan2.2-MoE on every VBench-2 and T2V-CompBench dimension reported in the table.
- SplitMoE reportedly reaches comparable validation loss with roughly 70% of the training steps of the dense baseline.
Surprising findings
- Partial prototype losses (only pull or only push) can underperform the variant without prototype guidance.
- Expert partitioning alone (SplitMoE w/o PG) gives limited gains over standard Wan2.2-MoE.
Limitations
- Relies on frozen visual features for prototype guidance.
- Sparse routing adds overhead relative to the dense model in speed measurements.
- Full SplitMoE is not best on several dimensions (e.g., Commonsense, Controllability, Physics vs. LongCat-Video, Wan2.2, LTX-2).
Experimental setup
Benchmarks: VBench-2.0 T2V-CompBench
Metrics: VBench-2 dimension scores (Creativity, Commonsense, Controll T2V-CompBench (Consistent Attribute, Interaction, Numeracy) validation diffusion loss training and inference seconds per iteration
Baselines: Dense Wan2.2-FT Wan2.2-MoE CogVideoX-1.5 Mochi HunyuanVideo LongCat-Video LTX-2 Wan2.2 OmniWeaving
Builds on: Wan2.2 DeepSeek-V3 ProMoE SRA2 flow matching
Closest papers by meaning
- Mixture of Nested Experts: Adaptive Processing of Visual Tokens (NeurIPS 2024 · similarity 0.88)
- Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization (NeurIPS 2024 · similarity 0.87)
- FastVID: Dynamic Density Pruning for Fast Video Large Language Models (NeurIPS 2025 · similarity 0.86)
- V-CAST: Video Curvature-Aware Spatio-Temporal Pruning for Efficient Video Large Language Models (NeurIPS 2026 · similarity 0.86)
- Expert Race: A Flexible Routing Strategy for Scaling Diffusion Transformer with Mixture of Experts (ICML 2025 · similarity 0.86)
- Towards Precise Scaling Laws for Video Diffusion Transformers (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.