Research map / Architectures and efficiency / Mixture of experts
Modality-Aware Expert Pruning for MoE-Based Multimodal Large Language Models
Donggeun Kim
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes modality-aware expert pruning for MoE-based multimodal LLMs to reduce expert memory that dominates model size.
Problem
Expert parameters dominate memory (over 90% of size) in MoE-based MLLMs such as InternVL3.5-30B-A3B and Qwen3-VL-30B-A3B, motivating expert pruning that accounts for multimodal inputs.
Approach
Expert pruning tailored to multimodal MoE models that accounts for modality differences in expert usage; the abstract is truncated so the precise mechanism is not stated.
Key results
- Expert parameters account for over 90% of model size in InternVL3.5-30B-A3B and Qwen3-VL-30B-A3B (54 GB out of ~58 GB in BF16).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| decompose expert importance by modality (text/vision split) (ablation) | MAEP on InternVL3.5-30B-A3B · ChartQA, 50% pruning | accuracy (%): 41.2 -> 76.7 | Improved |
| use low-attention vision tokens instead of full vision tokens as pruning signal (ablation) | MAEP on InternVL3.5-30B-A3B · ChartQA, 50% pruning | accuracy (%): 38.8 -> 76.7 | Improved |
| use low-attention instead of high-attention (top 20%) vision tokens (ablation) | MAEP on InternVL3.5-30B-A3B · ChartQA, 50% pruning | accuracy (%): 54.5 -> 76.7 | Improved |
| use high-attention vision tokens at top 2/5/10% (ablation) | MAEP on InternVL3.5-30B-A3B · 50% pruning | average retention: collapse (below 30% retention) | Hurt |
| use global cross-layer ranking instead of uniform per-layer budget (ablation) | MAEP on InternVL3.5-30B-A3B · ChartQA, 50% pruning | accuracy (%): 72.1 -> 76.7 | Improved |
| add ChartNet samples to calibration (7:3 mix) (ablation) | MAEP on Qwen3-VL-30B-A3B · ChartQA, 50% pruning | accuracy (%): 69.6 -> 83.5 | Improved |
| reduce calibration set from 1K to 100 samples (ablation) | MAEP on Qwen3-VL-30B-A3B · 50% pruning, four benchmarks | accuracy: ChartQA 69.6 -> 68.2; MMStar 60.2 -> 58.7 | No clear effect |
| raise low-attention quantile q from 2% toward 30% (ablation) | MAEP on InternVL3.5-30B-A3B · TextVQA, 50% pruning | accuracy | Hurt |
| increase vision-side weight alpha to 0.9 (ablation) | MAEP on InternVL3.5-30B-A3B · 50% pruning, avg retention | retention (%): 89.1 (alpha 0.7) -> 65.1 (alpha 0.9) | Hurt |
| increase vision-side weight alpha to 0.7 or above (ablation) | MAEP on Qwen3-VL-30B-A3B · 25% pruning, avg retention | retention (%): 97.1 (alpha 0.5) -> 29.0 (alpha 0.7) | Hurt |
| use MAEP instead of REAP at higher pruning ratio | MAEP on three MoE-MLLMs · 11 benchmarks avg retention | retention (%): InternVL3.5 88.5 -> 91.2; Gemma4 80.9 -> 86.7 at 50% | Improved |
| use MAEP instead of REAP on text-only tasks | MAEP on Qwen3-VL-30B-A3B · IFEval and ARC-C, 50% pruning | accuracy (%): IFEval 73.8 -> 69.3; ARC-C 85.9 -> 83.1 | Hurt |
Limitations
- Only the abstract is available; full method and results are unknown.
Closest papers by meaning
- LightMoE: Reducing Mixture-of-Experts Redundancy through Expert Replacing (NeurIPS 2026 · similarity 0.91)
- Pruning and Distilling Mixture-of-Experts into Dense Language Models (NeurIPS 2026 · similarity 0.91)
- FlexMoE: One-for-All Nested Intra-Expert Pruning for MoE Language Models (NeurIPS 2026 · similarity 0.91)
- Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations (NeurIPS 2025 · similarity 0.90)
- Mixture Compressor for Mixture-of-Experts LLMs Gains More (ICLR 2025 · similarity 0.90)
- DiEP: Adaptive Mixture-of-Experts Compression through Differentiable Expert Pruning (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
