Research map / Architectures and efficiency / Mixture of experts

Routing Experts: Learning to Route Dynamic Experts in Existing Multi-modal Large Language Models

ICLR 2025 · Architectures and efficiency › Mixture of experts · approach: Expert routing and specialization

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Routing Experts (RoE), which turns existing MLLMs (LLaVA-1.5, LLaVA-HR, VILA) into dynamic-depth mixtures of experts by routing each layer to either the frozen-then-tuned original layer or a lightweight adapter skip, trained with a difficulty-weighted sparsity regularizer and routing tokens.

Problem

Existing MLLMs are computationally expensive due to many visual tokens, and well-trained dense MLLMs resist layer skipping for dynamic inference; prior MoE-MLLMs need new architectures trained from scratch.

Approach

Treat each layer of an existing MLLM as an expert; learned per-example routers choose between the original layer and a low-rank adapter skip connection (to bridge the feature gap), with a loss-weighted structure sparsity regularizer and per-question routing tokens to align multi-turn training with per-example inference.

Key results

• RoE-LLaVA achieves 63.8 average accuracy over the five traditional VL benchmarks at 6.77 samples/s, versus MoE-LLaVA-2.7B×4 at 60.6 and 4.95 samples/s.
• RoE-VILA$_{30\%}$ skips 23.44% of layers with only about a 1.38% average performance drop and ~24.5% faster inference.
• Training RoE-VILA takes about 93.6 A800 GPU hours versus about 6447.4 GPU hours for training VILA from scratch.

Measured effects reported in the paper

Surprising findings

• A direct router added to LLaVA-1.5 barely skips layers (about 1.1% average skip rate, 0.01% on GQA), and adds about 7% latency.
• RoE-LLaVA-HR is sensitive to skipping: going from 7.68% to 12.02% skip rate drops average accuracy by about 4.3 points.

Limitations

• Sensitivity to skip rate varies by base model; LLaVA-HR degrades noticeably at higher skip rates.
• Speedup gains are modest (about 2-25%) and some benchmarks show small drops relative to dense baselines.
• Evaluated only on three LLaVA-family/VILA models with Vicuna-7B, so generality to other MLLM architectures is untested.

Experimental setup

Datasets: LLaVA-1.5 665k instruction data VQAv2 GQA ScienceQA VizWiz TextVQA POPE MME MMBench SEED-Bench MM-Vet

Benchmarks: VQAv2 GQA ScienceQA-IMG VizWiz TextVQA POPE MME MMBench SEED-Bench MM-Vet

Metrics: accuracy samples per second (speed) MME perception score MM-Vet score actual skip rate

Baselines: LLaVA-1.5 VILA LLaVA-HR MoE-LLaVA-1.6B×4 MoE-LLaVA-2.7B×4 Qwen-VL Qwen-VL-Chat

Builds on: Mixture of Experts LLaVA-1.5 Mixture-of-Depths LoRA-style adapters MoE-LLaVA

Closest papers by meaning

• Layerwise Recurrent Router for Mixture-of-Experts (ICLR 2025 · similarity 0.91)
• Universal Model Routing for Efficient LLM Inference (ICLR 2026 · similarity 0.90)
• Learning to Route Among Specialized Experts for Zero-Shot Generalization (ICML 2024 · similarity 0.90)
• Lookahead Routing for Large Language Models (NeurIPS 2025 · similarity 0.89)
• Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts (NeurIPS 2026 · similarity 0.89)
• dMoE: dLLMs with Learnable Block Experts (NeurIPS 2026 · similarity 0.89)