Research map / Architectures and efficiency / Mixture of experts
Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extracts MoE-style expert subnetworks from pretrained DeiT vision transformers post-training by HDBSCAN-clustering MLP activations, variance-based neuron selection, and cosine-similarity routing, then fine-tunes with distillation.
Problem
Building efficient MoE variants of vision transformers usually requires costly retraining or training from scratch; the paper asks whether experts can be extracted from already-trained ViTs to cut compute and memory.
Approach
Cluster token-wise MLP hidden activations with HDBSCAN (number of experts chosen from data), pick neurons per cluster by within-cluster variance, map cluster means back to the input space for cosine-similarity routing, and fine-tune briefly with knowledge distillation.
Key results
- DeiT-B-MoEE reduces MACs by 36.3% and parameters by 32.4% while reaching 80.12% Top-1 after fine-tuning versus 81.73% for DeiT-B.
- Before fine-tuning, DeiT-B-MoEE retains 68.54% accuracy; the MoEfication reimplementation retains 57.20%.
- Ablation on DeiT-B: HDBSCAN alone gives 75.36% Top-1; adding variance-based extraction and input-based routing reaches 80.12%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use data-driven expert extraction instead of MoEfication | DeiT-B · ImageNet-1k | Top-1 accuracy: 77.63 -> 80.12 | Helped |
| Use data-driven expert extraction instead of MoEfication | DeiT-B · ImageNet-1k | MACs reduction: -34.8% -> -36.3% | Helped |
| Add HDBSCAN clustering to random expert extraction (ablation) | DeiT-B-MoEE · ImageNet-1k | Top-1 accuracy: 58.64 -> 75.36 | Helped |
| Add variance-based neuron selection (ablation) | DeiT-B-MoEE with HDBSCAN clustering · ImageNet-1k | Top-1 accuracy: 75.36 -> 77.68 | Helped |
| Add input-based routing by similarity to cluster mean (ablation) | DeiT-B-MoEE with HDBSCAN and variance extraction · ImageNet-1k | Top-1 accuracy: 77.68 -> 80.1 | Helped |
| Use HDBSCAN rather than DBSCAN, OPTICS, K-Means, or BIRCH (ablation) | Expert extraction · ImageNet-1k | Top-1 accuracy | Helped |
| Prioritize components by activation variance rather than mean activation magnitude (ablation) | Expert extraction · ImageNet-1k | Top-1 accuracy | Helped |
| Use cosine similarity rather than Euclidean distance for expert routing (ablation) | Expert extraction · ImageNet-1k | Top-1 accuracy | No clear effect |
| Use about 100,000 tokens for extraction rather than very low sample sizes (ablation) | Expert extraction · ImageNet-1k | Variance between expert configurations: approximately 100,000 tokens | Helped |
Surprising findings
- Expert formation via HDBSCAN appears only in layers 6 through 11; earlier layers yield no distinct clusters and are left unchanged.
- Routing from the extracted experts is reasonably balanced without any load-balancing loss, unlike the MoEfication reimplementation.
Limitations
- Accuracy before fine-tuning is not sufficient for direct deployment (e.g., 55-68% retention).
- Cluster assignment uses a spherical-cluster approximation via mean input vectors, which can introduce routing errors that rely on fine-tuning to fix.
- Only evaluated on DeiT-style ViTs and ImageNet-1k in the main results.
Experimental setup
Datasets: ImageNet-1k CIFAR-100
Benchmarks: ImageNet-1k CIFAR-100
Metrics: Top-1 accuracy accuracy retention MACs parameter count routing distribution
Baselines: DeiT-T DeiT-S DeiT-B ViT-MoEfication (reimplementation) DBSCAN OPTICS K-Means BIRCH
Builds on: MoEfication HDBSCAN DeiT
Closest papers by meaning
- Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization (NeurIPS 2024 · similarity 0.90)
- From Sparse to Soft Mixtures of Experts (ICLR 2024 · similarity 0.90)
- A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts (ICML 2024 · similarity 0.89)
- Mixture of Nested Experts: Adaptive Processing of Visual Tokens (NeurIPS 2024 · similarity 0.89)
- Dense Backpropagation Improves Training for Sparse Mixture-of-Experts (NeurIPS 2025 · similarity 0.89)
- Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversion (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.