AI Paper Atlas

Research map / Architectures and efficiency / Mixture of experts

Modality-Aware Expert Pruning for MoE-Based Multimodal Large Language Models

Donggeun Kim

NeurIPS 2026 · Architectures and efficiency › Mixture of experts · approach: Expert routing and specialization · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes modality-aware expert pruning for MoE-based multimodal LLMs to reduce expert memory that dominates model size.

Problem

Expert parameters dominate memory (over 90% of size) in MoE-based MLLMs such as InternVL3.5-30B-A3B and Qwen3-VL-30B-A3B, motivating expert pruning that accounts for multimodal inputs.

Approach

Expert pruning tailored to multimodal MoE models that accounts for modality differences in expert usage; the abstract is truncated so the precise mechanism is not stated.

Key results

Measured effects reported in the paper

What was changedWhereMeasuredEffect
decompose expert importance by modality (text/vision split) (ablation)MAEP on InternVL3.5-30B-A3B · ChartQA, 50% pruningaccuracy (%): 41.2 -> 76.7Improved
use low-attention vision tokens instead of full vision tokens as pruning signal (ablation)MAEP on InternVL3.5-30B-A3B · ChartQA, 50% pruningaccuracy (%): 38.8 -> 76.7Improved
use low-attention instead of high-attention (top 20%) vision tokens (ablation)MAEP on InternVL3.5-30B-A3B · ChartQA, 50% pruningaccuracy (%): 54.5 -> 76.7Improved
use high-attention vision tokens at top 2/5/10% (ablation)MAEP on InternVL3.5-30B-A3B · 50% pruningaverage retention: collapse (below 30% retention)Hurt
use global cross-layer ranking instead of uniform per-layer budget (ablation)MAEP on InternVL3.5-30B-A3B · ChartQA, 50% pruningaccuracy (%): 72.1 -> 76.7Improved
add ChartNet samples to calibration (7:3 mix) (ablation)MAEP on Qwen3-VL-30B-A3B · ChartQA, 50% pruningaccuracy (%): 69.6 -> 83.5Improved
reduce calibration set from 1K to 100 samples (ablation)MAEP on Qwen3-VL-30B-A3B · 50% pruning, four benchmarksaccuracy: ChartQA 69.6 -> 68.2; MMStar 60.2 -> 58.7No clear effect
raise low-attention quantile q from 2% toward 30% (ablation)MAEP on InternVL3.5-30B-A3B · TextVQA, 50% pruningaccuracyHurt
increase vision-side weight alpha to 0.9 (ablation)MAEP on InternVL3.5-30B-A3B · 50% pruning, avg retentionretention (%): 89.1 (alpha 0.7) -> 65.1 (alpha 0.9)Hurt
increase vision-side weight alpha to 0.7 or above (ablation)MAEP on Qwen3-VL-30B-A3B · 25% pruning, avg retentionretention (%): 97.1 (alpha 0.5) -> 29.0 (alpha 0.7)Hurt
use MAEP instead of REAP at higher pruning ratioMAEP on three MoE-MLLMs · 11 benchmarks avg retentionretention (%): InternVL3.5 88.5 -> 91.2; Gemma4 80.9 -> 86.7 at 50%Improved
use MAEP instead of REAP on text-only tasksMAEP on Qwen3-VL-30B-A3B · IFEval and ARC-C, 50% pruningaccuracy (%): IFEval 73.8 -> 69.3; ARC-C 85.9 -> 83.1Hurt

Limitations

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy