Research map / Vision-language and multimodal / Multimodal LLMs
DTP: Delta-Guided Two Stage Pruning for Mamba-based Multimodal Large Language Models
Seong Yeol Park, Min Jung Kwon, XIANGHUA PIAO, Yeong Hyeon Gu
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DTP, a delta-guided two-stage visual token pruning method that progressively removes redundant visual tokens in Mamba-based multimodal LLMs to cut prefill cost.
Problem
Mamba-based multimodal LLMs are slowed by redundant visual tokens inflating inference cost, with prefill accounting for most inference time.
Approach
Progressively prunes visual tokens in two stages, selective pruning at an early layer followed by complete pruning at a late layer, with pruning guided by the Mamba delta (step-size) parameter.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use delta_t as token importance instead of C_t (ablation) | DTP on Cobra · TextVQA (~50% FLOPs) | accuracy: 44.9 -> 56.1 | Improved |
| use delta_t instead of y_t (ablation) | DTP on Cobra · TextVQA | accuracy: 48.0 -> 56.1 | Improved |
| restrict top-k pruning to visual tokens only (do not prune text tokens) (ablation) | DTP on Cobra · TextVQA | accuracy: 10.3 -> 56.1 | Improved |
| top-k delta selection instead of random visual token selection (ablation) | DTP on Cobra · TextVQA | accuracy: 53.7 -> 56.1 | Improved |
| add complete pruning of visual tokens at late layer 45 (ablation) | DTP on Cobra · GQA/TextVQA/VizWiz (~50% FLOPs) | accuracy and FLOPs: FLOPs 1.26 -> 0.97; GQA 61.5 -> 61.4 | Improved |
| add complete late pruning (ablation) | DTP on RoboMamba · GQA/POPE/OKVQA | accuracy and FLOPs: FLOPs 0.44 -> 0.34; GQA 55.1 -> 54.9 | Improved |
| DTP two-stage delta-guided pruning vs FastV/DART at ~70% FLOPs | Cobra · six-benchmark average | score: 65.3 (FastV), 65.2 (DART) -> 65.8 | Improved |
| DTP at ~50% FLOPs vs other pruning | Cobra · six-benchmark average | score: 64.9 vs FastV 64.7, DART 64.3, VTW 54.0 | Mixed |
| fix VTW withdrawal layer at 32 instead of 45 | VTW on Cobra · six-benchmark average | score: 65.7 -> 54.0 | Hurt |
| DTP pruning on RoboMamba (256 tokens) at ~50% FLOPs | RoboMamba · five-benchmark average | score: 67.0 -> 65.9 | Mixed |
| apply DTP to reduce prefill | Cobra · POPE latency | prefill ms: 98.04 -> 61.54 | Improved |
Closest papers by meaning
- Fit and Prune: Fast and Training-free Visual Token Pruning for Multi-modal Large Language Models (AAAI 2025 · similarity 0.92)
- Hi-Lo Prune: Look at What You'll Lose before Pruning with Hierarchical Token Selection (CVPR 2026 · similarity 0.91)
- ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming (AAAI 2025 · similarity 0.91)
- ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · similarity 0.90)
- Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs (NeurIPS 2025 · similarity 0.90)
- EfficientLLaVA: Generalizable Auto-Pruning for Large Vision-language Models (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
