Research map / Vision-language and multimodal / Multimodal LLMs
Recoverable Compression: A Multimodal Vision Token Recovery Mechanism Guided by Text Information
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a training-free, text-guided dynamic visual token recovery mechanism for LLaVA-1.5 that prunes visual tokens by CLS-token scores, recovers text-relevant tokens, and KNN-merges the rest, reaching ~10% token usage.
Problem
Visual token pruning for multimodal LLMs based only on visual importance can discard question-relevant information, while inference cost grows with the number of visual tokens.
Approach
Combine a CLS-token visual score with a question-text similarity score (through an MLP projection) to recover pruned but question-relevant visual tokens, using LOF-based dynamic outlier selection instead of fixed thresholds, and then KNN-merge the remaining background tokens back in order.
Key results
- Training-free version reaches 69.01 on ScienceQA and 55.51 on TextVQA using ~9.7-9.9% of visual tokens, beating LLaVA-PruMerge (68.52 / 53.51) in the same training-free setting.
- On LLaVA1.5-7B FP16 at A100, FLOPs drop from 8.5T to 1.5T and prefill time from 30.3 ms to 9.2 ms.
- Fine-tuned variant reaches 68.72 on ScienceQA and 56.16 on TextVQA, but underperforms LLaVA-PruMerge+ and CrossGET on VQAv2, POPE and MMBench.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply text-guided visual token recovery instead of visual-score-only pruning (ablation) | LLaVA1.5-7B · ScienceQA | score: 68.72 -> 68.91 | Helped |
| apply text-guided visual token recovery instead of visual-score-only pruning (ablation) | LLaVA1.5-7B · TextVQA | score: 54.94 -> 55.33 | Helped |
| apply text-guided visual token recovery instead of visual-score-only pruning (ablation) | LLaVA1.5-7B · MME | score: 1190.3 -> 1196.9 | Helped |
| use dynamic scale filtering instead of a fixed threshold (ablation) | LLaVA1.5-7B · ScienceQA | score: 68.47 -> 68.57 | Helped |
| use dynamic scale filtering instead of a fixed threshold (ablation) | LLaVA1.5-7B · TextVQA | score: 54.20 -> 54.60 | Helped |
| use dynamic scale filtering instead of a fixed threshold (ablation) | LLaVA1.5-7B · MME | score: 1139.2 -> 1146.9 | Helped |
| add KNN secondary recovery of the remaining tokens (ablation) | LLaVA1.5-7B with text-guided recovery · ScienceQA | score: 68.91 -> 69.01 | Helped |
| add KNN secondary recovery of the remaining tokens (ablation) | LLaVA1.5-7B with text-guided recovery · TextVQA | score: 55.33 -> 55.51 | Helped |
| add KNN secondary recovery of the remaining tokens (ablation) | LLaVA1.5-7B with text-guided recovery · MME | score: 1196.9 -> 1284.9 | Helped |
| compress visual tokens to around 10% using the proposed method rather than the original model | LLaVA1.5-7B · ScienceQA | score: 68.40 -> 69.01 | Helped |
| compress visual tokens to around 10% using the proposed method rather than the original model | LLaVA1.5-7B · TextVQA | score: 58.20 -> 55.51 | Hurt |
| compress visual tokens to around 10% using the proposed method rather than the original model | LLaVA1.5-7B · MME | score: 1476.90 -> 1284.90 | Hurt |
Surprising findings
- Using about 50% of tokens gives slightly lower ScienceQA accuracy (68.52) than using 10% (69.01), so more tokens does not always help.
- Performance drops on VQAv2, POPE and MMBench relative to the full LLaVA-1.5 model, which the authors attribute to needing raw visual features.
Limitations
- Performance still lags the full model on several benchmarks (VQAv2, POPE, MMBench), per the authors.
- Only single-round VQA is addressed; multi-round dialogue is left for future work.
- Ablations use a small set of datasets and do not report variance.
Experimental setup
Datasets: ScienceQA TextVQA MME VQAv2 POPE MMBench
Benchmarks: ScienceQA TextVQA MME VQAv2 POPE MMBench DocVQA ChartQA
Metrics: accuracy MME score FLOPs prefill time memory
Baselines: LLaVA-PruMerge LLaVA-PruMerge+ CrossGET Chat-UniVi ToMe BLIP-2 Qwen-VL InstructBLIP
Builds on: LLaVA-1.5 LLaVA-PruMerge Local Outlier Factor ToMe LoRA
Closest papers by meaning
- Efficient Large Multi-modal Models via Visual Context Compression (NeurIPS 2024 · similarity 0.92)
- Rethinking Token Reduction for Large Vision-Language Models (CVPR 2026 · similarity 0.91)
- Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.91)
- What Kind of Visual Tokens Do We Need? Training-Free Visual Token Pruning for Multi-Modal Large Language Models from the Perspective of Graph (AAAI 2025 · similarity 0.91)
- EvoComp: Learning Visual Token Compression for Multimodal Large Language Models via Semantic-Guided Evolutionary Labeling (CVPR 2026 · similarity 0.91)
- Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.