Research map / Vision-language and multimodal / Multimodal LLMs
VisionZip: Longer is Better but Not Necessary in Vision Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VisionZip, a text-agnostic method that compresses visual tokens from CLIP/SigLIP vision encoders (dominant token selection by attention plus contextual token merging) before the LLM, with optional projector tuning, to cut VLM cost while preserving accuracy.
Problem
Vision-language models use very long visual token sequences (e.g., 576 to 2880 tokens), which makes inference costly, and existing text-relevant token pruning methods (FastV, SparseVLM) select tokens poorly and fail in multi-turn dialogue.
Approach
Visual tokens from the vision encoder contain much redundancy: attention concentrates on a few dominant tokens. VisionZip keeps dominant tokens chosen by CLS (or mean) attention from the encoder, merges the remaining tokens into a few contextual tokens by key similarity, and optionally fine-tunes only the projector, all outside the LLM and independent of the text query.
Key results
- On LLaVA-1.5 7B with 64 tokens (88.9% reduction), VisionZip with projector tuning retains 95.2% of vanilla average performance, versus 75.6% for FastV and 85.8% for SparseVLM.
- On LLaVA-NeXT 7B, VisionZip at 160 tokens keeps 92.0% of performance training-free and reduces prefilling time from 218ms to 27.8ms (7.8x) on POPE.
- On Video-LLaVA with 136 tokens, VisionZip achieves 93.2% average performance versus 86.5% for SparseVLM.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Select informative visual tokens with VisionZip instead of using SparseVLM | LLaVA-1.5 7B with 192 retained tokens · LLaVA-1.5 image understanding benchmarks | average relative accuracy: 96.4% -> 98.5% | Helped |
| Select informative visual tokens with VisionZip instead of using SparseVLM | LLaVA-1.5 7B with 64 retained tokens · LLaVA-1.5 image understanding benchmarks | average relative accuracy: 85.8% -> 94.0% | Helped |
| Select informative visual tokens with VisionZip instead of using SparseVLM | LLaVA-NeXT 7B with 160 retained tokens · LLaVA-NeXT image understanding benchmarks | average relative accuracy: 86.4% -> 92.0% | Helped |
| Select informative visual tokens with VisionZip instead of using SparseVLM | Video-LLaVA with 136 retained tokens · Video QA benchmarks | average relative performance: 86.5% -> 93.2% | Helped |
| Reduce visual tokens to 160 using VisionZip | LLaVA-NeXT 7B · POPE | total inference time: 2293s -> 756s | Helped |
| Reduce visual tokens to 160 using VisionZip | LLaVA-NeXT 7B · POPE | prefilling time: 218ms -> 27.8ms | Helped |
| Add efficient projector fine-tuning with 1/10 of the LLaVA-1.5 dataset (ablation) | VisionZip on LLaVA-1.5 7B with 64 retained tokens · LLaVA-1.5 image understanding benchmarks | average relative accuracy: 94.0% -> 95.2% | Helped |
| Add efficient projector fine-tuning with 1/10 of the LLaVA-1.5 dataset (ablation) | VisionZip on LLaVA-NeXT with 160 retained tokens · LLaVA-NeXT image understanding benchmarks | average relative accuracy: 92.0% -> 95.5% | Helped |
| Mask the 50 highest-attention tokens before SparseVLM selects 64 (ablation) | SparseVLM · TextVQA | accuracy: 51.1 -> 46.4 | Hurt |
| Give SparseVLM the top 128 VisionZip-selected tokens before it filters to 64 (ablation) | SparseVLM · TextVQA | accuracy: 51.1 -> 52.5 | Helped |
| Retain fewer visual tokens using VisionZip (ablation) | LLaVA-1.5 7B · LLaVA-1.5 image understanding benchmarks | average relative accuracy: 98.5% -> 97.6% -> 94.0% | Hurt |
| Retain fewer visual tokens using VisionZip (ablation) | LLaVA-NeXT 7B · LLaVA-NeXT image understanding benchmarks | average relative accuracy: 97.6% -> 95.0% -> 92.0% | Hurt |
Surprising findings
- Pruning visual tokens sometimes improves benchmark scores (e.g., MM-Vet and MMMU), which the authors attribute to redundant tokens acting as noise.
- Using SparseVLM to select from all 576 tokens lowers TextVQA accuracy, whereas feeding it VisionZip's 128 selected tokens raises it, suggesting text-relevant selection misaligns with encoder-aggregated tokens.
Limitations
- Projector fine-tuning needed to close the gap at aggressive compression (64 tokens)
- Non-CLS encoder (SigLIP) variant handled by a different selection rule described only in the appendix
Experimental setup
Datasets: LLaVA-1.5 instruction tuning data TextVQA validation
Benchmarks: GQA MMBench MME POPE ScienceQA VQAv2 TextVQA MMMU SEED-Bench MM-Vet LLaVA-Bench TGIF-QA
Metrics: accuracy relative performance vs. vanilla total inference time prefilling latency GPU memory ChatGPT score (video)
Baselines: FastV SparseVLM Vanilla LLaVA-1.5 Video-LLaVA
Builds on: FastV SparseVLM LLaVA-1.5 LLaVA-NeXT
Closest papers by meaning
- Rethinking Token Reduction for Large Vision-Language Models (CVPR 2026 · similarity 0.90)
- MMLongBench: Benchmarking Long-Context Vision-Language Models Effectively and Thoroughly (NeurIPS 2025 · similarity 0.90)
- ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · similarity 0.89)
- LLMC+: Benchmarking Vision-Language Model Compression with a plug-and-play Toolkit (AAAI 2026 · similarity 0.89)
- iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models (ICLR 2026 · similarity 0.89)
- Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.