Research map / Architectures and efficiency / Mixture of experts
MoE Jetpack: From Dense Checkpoints to Adaptive Mixture of Experts for Vision Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Fine-tunes pre-trained dense ViT/ConvNeXt checkpoints into sparse MoE models via importance-based checkpoint recycling and a hyperspherical cross-attention routing layer (SpheroMoE), accelerating convergence and improving accuracy on image classification.
Problem
Training MoE models from scratch needs large data and compute, and public model hubs mostly host dense checkpoints, so MoE adoption is limited. The paper asks whether dense checkpoints can speed up and improve MoE fine-tuning for vision.
Approach
Checkpoint recycling splits a dense predecessor's MLPs into multiple experts using weight-selection strategies (importance-based sampling by default), and the SpheroMoE layer adds layer-normalized inputs, L2-normalized queries with cross-attention dispatch, learnable softmax temperature with expert noise and expert dropout, and a core/universal dual-path structure.
Key results
- MoE Jetpack (V-JetMoE-T, ViT-T-scale, ~1.1 GFLOPs) reaches 79.9% ImageNet-1k top-1 vs 77.1% for Soft MoE trained from scratch and 73.9% for dense ViT-T.
- On CIFAR-100, MoE Jetpack reaches 88.4% vs 75.9% for Soft MoE; on Pets it reaches 84.3% vs 45.5% for Soft MoE.
- Convergence: MoE Jetpack reaches ~77% ImageNet top-1 about 2x faster than training from scratch and reaches 76% CIFAR-100 about 8x faster.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| fine-tune a dense ImageNet-21k checkpoint with MoE Jetpack instead of training Soft MoE from scratch | ViT-T · ImageNet-1K | accuracy: 77.1 -> 79.9 | Helped |
| fine-tune a dense ImageNet-21k checkpoint with MoE Jetpack instead of training Soft MoE from scratch | ViT-T · CIFAR-100 | accuracy: 75.9 -> 88.4 | Helped |
| fine-tune a dense ImageNet-21k checkpoint with MoE Jetpack instead of training Soft MoE from scratch | ConvNeXt-F · ImageNet-1K | accuracy: 79.1 -> 80.5 | Helped |
| add checkpoint recycling to Soft MoE (ablation) | ViT-T · CIFAR-100 | accuracy: 75.9 -> 84.7 | Helped |
| add SpheroMoE to Soft MoE with checkpoint recycling (ablation) | ViT-T · Mean across ImageNet, CIFAR-100, Flowers | mean accuracy: 84.8 -> 87.9 | Helped |
| replace Sparse Upcycling with importance-based checkpoint sampling (ablation) | ViT-T with SpheroMoE · ImageNet-1K | accuracy: 79.1 -> 79.9 | Helped |
| replace importance-based sampling with graph partitioning (ablation) | ViT-T with SpheroMoE · ImageNet-1K | accuracy: 79.9 -> 79.8 | Hurt |
| replace importance-based sampling with uniform selection (ablation) | ViT-T with SpheroMoE · ImageNet-1K | accuracy: 79.9 -> 79.6 | Hurt |
| replace importance-based sampling with random sampling (ablation) | ViT-T with SpheroMoE · ImageNet-1K | accuracy: 79.9 -> 79.5 | Hurt |
| use SpheroMoE layers 5–12 rather than layers 7–12 (ablation) | V-JetMoE-T · CIFAR-100 | accuracy: 88.4 -> 86.7 | Hurt |
| increase SpheroMoE layers from 7–12 to 9–12 (ablation) | V-JetMoE-T · CIFAR-100 | accuracy: 88.4 -> 87.8 | Hurt |
| use SpheroMoE layers 11–12 rather than layers 7–12 (ablation) | V-JetMoE-T · CIFAR-100 | accuracy: 88.4 -> 87.4 | Hurt |
Surprising findings
- Sparse upcycling with the same SpheroMoE layer gives 79.1% ImageNet vs 79.9% for importance-based checkpoint recycling, so the recycling strategy adds only a small gain over copying MLPs.
- Dense ViT-T trained from ImageNet-21k pre-training is reported at 75.6% ImageNet, below the MoE Jetpack result, and the paper reports that expert initialization from dense weights gives large gains on small datasets.
Limitations
- Performance depends on the quality of the pre-trained dense checkpoints.
- Evaluated only on image classification; generalization to NLP and reinforcement learning is untested.
- Ablations and the configuration study are restricted to a few datasets and the ViT-T scale.
Experimental setup
Datasets: ImageNet-21k ImageNet-1K CIFAR-10 CIFAR-100 Flowers Pets STL-10 Food-101 DTD
Benchmarks: ImageNet-1K CIFAR-10 CIFAR-100 Flowers Pets STL-10 Food-101 DTD
Metrics: top-1 accuracy mean accuracy convergence epochs
Baselines: Soft MoE Dense ViT (ImageNet-1k from scratch) Dense ViT (ImageNet-21k pre-trained) Sparse Upcycling Soft MoE-S
Builds on: Soft MoE Sparse Upcycling Weight Selection Vision Transformer ConvNeXt
Closest papers by meaning
- Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization (NeurIPS 2024 · similarity 0.87)
- MoE-CAP: Benchmarking Cost, Accuracy and Performance of Sparse Mixture-of-Experts Systems (NeurIPS 2025 · similarity 0.87)
- Scaling Beyond the GPU Memory Limit for Large Mixture-of-Experts Model Training (ICML 2024 · similarity 0.86)
- Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient (ICML 2025 · similarity 0.86)
- Dense Backpropagation Improves Training for Sparse Mixture-of-Experts (NeurIPS 2025 · similarity 0.86)
- MoE++: Accelerating Mixture-of-Experts Methods with Zero-Computation Experts (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.