Research map / Architectures and efficiency / Mixture of experts
Mixture of Nested Experts: Adaptive Processing of Visual Tokens
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Mixture of Nested Experts (MoNE), which routes visual tokens via a learned router and Expert Preferred Routing to nested ViT sub-models of increasing width, cutting inference FLOPs about 2-3x on images and videos at matched accuracy.
Problem
ViT and ViViT models process all visual tokens with equal compute despite high spatial and temporal redundancy, while MoE adds parameters and keeps per-token compute fixed.
Approach
Nested experts (weight-slices of the same network, following MatFormer) are used as MoE experts with no extra parameters; a router assigns tokens to nested experts under capacity constraints via Expert Preferred Routing, so redundant tokens use cheaper experts and a single model can serve multiple compute budgets.
Key results
- On SSv2, MoNE reaches 64.6 accuracy at 162.8 FLOPs versus 64.4 for the ViViT baseline at 376.3 FLOPs (about 2.3x reduction).
- MoNE matches baseline ImageNet-21k performance with roughly 2x FLOP reduction and outperforms MatViT nested models and MoD, especially at low FLOPs.
- A single MoNE model trained with capacities sampled from 0.15 to 0.95 adapts to a range of inference-time budgets on SSv2 at isoFLOPs training cost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use adaptive MoNE token routing instead of processing all tokens with the full ViViT baseline | ViViT Factorized Encoder B/16 · Something-Something-v2 8x1 test | accuracy: 64.4 -> 64.6 | Helped |
| Use adaptive MoNE token routing instead of the full ViViT baseline | ViViT Factorized Encoder B/16 · Something-Something-v2 8x1 test | FLOPs: 376.3 -> 162.8 | Helped |
| Use heuristic MoNE capacity allocation instead of proportionate allocation | MoNE · Something-Something-v2 | accuracy: 64.3 -> 64.6 | Helped |
| Use heuristic MoNE capacity allocation instead of uniform allocation | MoNE · Something-Something-v2 | accuracy: 64.6 -> 64.6 | No clear effect |
| Use heuristic MoNE capacity allocation instead of MoD Greedy allocation | MoNE · Something-Something-v2 | accuracy: 63.9 -> 64.6 | Helped |
| Use heuristic MoNE capacity allocation instead of Greedy allocation | MoNE · Something-Something-v2 | accuracy: 64.2 -> 64.6 | Helped |
| Place the router at the first layer rather than a later layer (ablation) | MoNE · Something-Something-v2 | performance | Helped |
| Use one router at the first layer rather than increasing the number of routers throughout the network (ablation) | MoNE · Something-Something-v2 | performance | Helped |
| Use a learned router instead of a random router (ablation) | MoNE · Something-Something-v2 | accuracy | Helped |
Surprising findings
- Adding more routers at multiple layers degrades performance; a single router at the first layer works best.
- Performance degrades more gracefully when adapting models across capacities on videos than on images.
Limitations
- Extension to autoregressive decoding in LLMs is non-trivial because the method is designed for encoder architectures.
- Learned router performance degrades under large train-to-eval capacity shifts, especially from high to low capacity on images.
Experimental setup
Datasets: ImageNet-21k Kinetics-400 Something-Something-v2
Benchmarks: ImageNet-21k Kinetics-400 Something-Something-v2
Metrics: precision@1 top-1 accuracy GFLOPs
Baselines: ViT ViViT MatViT Mixture of Depths
Builds on: MatFormer MatViT Mixture of Depths ViViT expert choice routing
Closest papers by meaning
- Efficient Data Driven Mixture-of-Expert Extraction from Trained Networks (CVPR 2025 · similarity 0.89)
- Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization (NeurIPS 2024 · similarity 0.89)
- Breaking the Uniformity Trap: Scaling Video Diffusion Model via SplitMoE (NeurIPS 2026 · similarity 0.88)
- From Sparse to Soft Mixtures of Experts (ICLR 2024 · similarity 0.87)
- Toward Efficient Inference for Mixture of Experts (NeurIPS 2024 · similarity 0.87)
- Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.