Research map / Vision-language and multimodal / Multimodal LLMs
Streamline Without Sacrifice - Squeeze out Computation Redundancy in LMM
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ProxyV, which runs heavy decoder-layer computation (attention and FFN) on a downsampled set of proxy vision tokens and updates full vision tokens via lightweight guided-update MLPs, cutting prefill FLOPs while keeping or improving fine-grained benchmark scores in decoder-only LMMs.
Problem
Decoder-only large multimodal models incur high prefill compute because every vision token passes through all self-attention and FFN layers of the LLM, and token-reduction methods risk discarding fine-grained visual information.
Approach
Instead of pruning or merging vision tokens, study computation-level redundancy: heavy attention/FFN ops are applied only to a small set of spatially downsampled proxy vision tokens, and each proxy token guides its r×r block of full vision tokens via a lightweight MLP update, with a non-spatial variant that composes with token reduction.
Key results
- ProxyV on Vicuna-1.5-7B from layer 16 reaches 102.4% of baseline Score_fine (55.94 vs 54.64) with 36% FLOPs and 31% time reduction in prefill.
- VisionZip and PyramidDrop, configured for similar efficiency, drop to 79.11 and 80.75 on RefCOCO versus 82.96 baseline, and reach BLEU 0.7218 and 0.7261 on document parsing versus 0.7991 for the baseline.
- Non-spatial ProxyV-ns combined with VisionZip reaches 54.83 Score_fine with 62% FLOPs and 65% time reduction.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply ProxyV from layer 12 | Vicuna-7B · Fine-grained benchmarks (Score_fine) | Score_fine: 54.64 -> 55.16 | Helped |
| Apply ProxyV from layer 16 | Vicuna-7B · Fine-grained benchmarks (Score_fine) | Score_fine: 54.64 -> 55.94 | Helped |
| Apply ProxyV at layer 16 or layer 20 | Vicuna-13B, Llama3-8B, Qwen2-7B, Phi3-3B, and InternLM2.5-7B · Fine-grained benchmarks (Score_fine) | Score_fine: 58.58 -> 58.95; 58.58 -> 59.22; 56.60 -> 56.87; 56.60 -> 57.49; 60.15 -> 60.55; 60.15 -> 61.44; 49.89 -> 50.28; 49.89 -> | Helped |
| Finetune while skipping vision-to-all attention from layer 0 (ablation) | Vicuna-7B · Fine-grained benchmarks (Score_fine relative) | Score_fine relative: 62.7% -> 87.2% | Helped |
| Finetune while skipping vision-to-all attention from layer 12 (ablation) | Vicuna-7B · Fine-grained benchmarks (Score_fine relative) | Score_fine relative: 94.5% -> 99.3% | Helped |
| Finetune while skipping vision-to-all attention from layer 16 (ablation) | Vicuna-7B · Fine-grained benchmarks (Score_fine relative) | Score_fine relative: 100.2% -> 99.9% | Hurt |
| Add lightweight vision-specific MLPs while skipping attention and FFNs from layer 0 (ablation) | Vicuna-7B · Fine-grained benchmarks (Score_fine relative) | Score_fine relative: 87.2% -> 65.1% | Hurt |
| Add lightweight vision-specific MLPs while skipping attention and FFNs from layer 12 (ablation) | Vicuna-7B · Fine-grained benchmarks (Score_fine relative) | Score_fine relative: 99.3% -> 99.3% | No clear effect |
| Add lightweight vision-specific MLPs while skipping attention and FFNs from layer 16 (ablation) | Vicuna-7B · Fine-grained benchmarks (Score_fine relative) | Score_fine relative: 99.9% -> 100.4% | Helped |
| Replace spatial constraints with non-spatial ProxyV (ablation) | Vicuna-7B · Fine-grained benchmarks (Score_fine) | Score_fine: 55.16 -> 54.85 | No clear effect |
| Combine non-spatial ProxyV with VisionZip (ablation) | Vicuna-7B · Fine-grained benchmarks (Score_fine) | Score_fine: 54.58 -> 54.83 | No clear effect |
| Use token reduction instead of ProxyV | Vicuna-7B · RefCOCO | RefCOCO score: 84.41 -> 79.11; 84.41 -> 80.75 | Hurt |
Surprising findings
- Skipping attention and FFN on vision tokens while adding lightweight vision-specific MLPs can improve fine-grained performance in deeper layers (L16 ATN+FFN reaches 100.4%).
- Vision-to-vision attention can be masked entirely from middle layers onward in several LLMs with little or no performance loss.
Limitations
- Gains are modest (around 1-2%) and efficiency gains shrink for some LLMs when applied later (e.g., Qwen2-7B 14% time reduction at layer 20).
- Guided-update module lacks local attention or convolution; left for future work.
- Evaluated mainly on LLaVA-NeXT-style models with CLIP-ViT-L at 336px; video and multi-image settings are not tested.
Experimental setup
Datasets: ShareGPT4V LLaVA-NeXT instruction tuning data DocStruct4M
Benchmarks: DocVQA ChartQA InfoVQA TextVQA OCRBench RefCOCO CCpdf document parsing MMBench SEED-Bench MMStar GQA MME
Metrics: average score on fine-grained OCR benchmarks (Score_fine) relative performance prefill FLOPs reduction prefill time reduction BLEU edit distance
Baselines: VisionZip PyramidDrop Baseline LLaVA-NeXT-style LMM (no reduction)
Builds on: LLaVA-NeXT Libra EE-MLLM
Closest papers by meaning
- Variation-aware Vision Token Dropping for Faster Large Vision-Language Models (CVPR 2026 · similarity 0.89)
- iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models (ICLR 2026 · similarity 0.89)
- LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token (ICLR 2025 · similarity 0.88)
- VisionTrim: Unified Vision Token Compression for Training-Free MLLM Acceleration (ICLR 2026 · similarity 0.88)
- VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation (NeurIPS 2024 · similarity 0.87)
- SCoRe: Salience-Coverage Reduction for Vision Token Pruning in Vision-Language Models (CVPR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.