Research map / Vision-language and multimodal / Multimodal LLMs

$\gamma-$MoD: Exploring Mixture-of-Depth Adaptation for Multimodal Large Language Models

ICLR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Training-free token reduction for MLLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Adapts mixture-of-depths routing to existing MLLMs (LLaVA, LLaVA-HR, Mini-Gemini-HD) by selecting redundant layers via attention-map rank (ARank), sharing one vision-language router, and masking question tokens from skipping, skipping ~37-58% of tokens with small accuracy loss.

Problem

Multimodal LLMs are computationally expensive at inference and training; directly converting dense layers to mixture-of-depths (token-skipping) layers in pretrained MLLMs causes large performance drops, and there is no practical guidance on which layers to convert.

Approach

Use the average rank of attention maps (ARank), estimated on a small batch of samples, to identify redundant layers to convert to MoD layers; add a single router shared across all MoD layers that routes both visual and text tokens, and a masked routing loss that prevents question tokens from being skipped during instruction tuning.

Key results

• With ARank-based deployment plus masked routing, LLaVA-HR skips 57.7% of tokens with average accuracy 57.6 vs 58.5 dense (-1.5%), cutting TFlops from 19.2 to 9.3.
• Training time of LLaVA-HR falls 31.0% (20.7h to 14.3h) and inference throughput rises 53.2% (4.7 to 7.2 samples/s) at the 0.5 routing ratio.
• Default MoD (all layers or hand-crafted interleaving) degrades LLaVA-HR considerably, e.g., TextVQA drops to 33.8 versus 67.1 for the dense model when all layers are converted.

Measured effects reported in the paper

Surprising findings

• Directly converting all layers to MoD layers drops TextVQA from 67.1 to 33.8 on LLaVA-HR, showing naive MoD adaptation is highly damaging.
• Masked routing learning reduces TFlops further (10.7 to 9.3) while also slightly improving accuracy.

Limitations

• Performance degrades noticeably on low-resolution LLaVA at high routing ratio (0.5 gives GQA 41.4 vs 62.0).
• Requires a calibration step to compute ARank and a fixed routing ratio chosen per model.

Experimental setup

Datasets: LCS-558K LLaVA-1.5 665k instruction data GQA SQA MMMU TextVQA POPE MME MMBench MM-Vet VQAv2 VizWiz

Benchmarks: GQA SQA MMMU TextVQA POPE MME MMB MM-Vet VQAv2 VizWiz SEED

Metrics: accuracy skip ratio TFlops training time inference throughput inference memory

Baselines: LLaVA-HR LLaVA-1.5 Mini-Gemini-HD MoE-LLaVA Default MoD Qwen-VL VILA InstructBLIP I-80B

Builds on: mixture of depths mixture of experts LLaVA-HR HRank

Closest papers by meaning

• Accelerating Multimodal Large Language Models by Searching Optimal Vision Token Reduction (CVPR 2025 · similarity 0.89)
• DenseMLLM: Standard Multimodal LLMs for Dense Prediction (ICML 2026 · similarity 0.88)
• Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.88)
• Learn from Downstream and Be Yourself in Multimodal Large Language Models Fine-Tuning (ICML 2025 · similarity 0.88)
• NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints (NeurIPS 2025 · similarity 0.88)
• IAA: Inner-Adaptor Architecture Empowers Frozen Large Language Model with Multimodal Capabilities (AAAI 2025 · similarity 0.87)