Research map / Architectures and efficiency / Attention and transformers
DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DeepStack, which splits visual tokens into a grid of N groups and injects each group residually into successive early LLM (and ViT) layers, boosting LMM performance on high-resolution benchmarks without adding context length.
Problem
Standard LMMs feed all visual tokens as a prefix into the first LLM layer, which greatly increases compute and memory cost, especially for high-resolution images and videos, and existing token compression sacrifices fine-grained detail.
Approach
Stack high-resolution visual tokens into groups aligned with successive early transformer layers (bottom-to-top) and add each group via a residual connection, so more visual information is conveyed without increasing the input context length; the same idea is applied to ViT layers.
Key results
- DeepStack-L-7B (Vicuna-7B) improves on LLaVA-1.5-7B by 4.2 on TextVQA, 11.0 on DocVQA, and 4.0 on InfoVQA with the same 576 context length.
- On Phi-3 ablations, DeepStack with high-resolution stacking raises the 7-benchmark average from 49.1 to 51.1 with no extra visual tokens.
- Fine-tuning the vision encoder combined with DeepStack reaches an average of 52.4 versus 48.5 for the baseline with fine-tuned encoder.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply DeepStack visual-token stacking across LLM layers | LLaVA-1.5-7B · DocVQA | accuracy: 28.1 -> 39.1 | Helped |
| Apply DeepStack visual-token stacking across LLM layers | LLaVA-1.5-7B · TextVQA | accuracy: 58.2 -> 62.4 | Helped |
| Apply DeepStack visual-token stacking across LLM layers | LLaVA-1.5-7B · InfoVQA | accuracy: 25.8 -> 29.8 | Helped |
| Apply DeepStack visual-token stacking across LLM layers | LLaVA-1.5-7B · MM-Vet | score: 30.5 -> 29.9 | Hurt |
| Stack high-resolution visual tokens instead of repeated original tokens (ablation) | DeepStack · Phi-3 evaluation suite | AVG: 49.1 -> 51.1 | Helped |
| Use spatially consistent 2d Spatial sampling rather than 2d Grid sampling (ablation) | DeepStack · Phi-3 evaluation suite | AVG: 49.0 -> 51.1 | Helped |
| Use spatially consistent 2d Spatial sampling rather than 1d Sequential sampling (ablation) | DeepStack · Phi-3 evaluation suite | AVG: 49.3 -> 51.1 | Helped |
| Fine-tune the vision encoder (ablation) | DeepStack · Phi-3 evaluation suite | AVG: 51.1 -> 52.4 | Helped |
| Fine-tune the vision encoder (ablation) | No token enhancement · Phi-3 evaluation suite | AVG: 49.1 -> 48.5 | Hurt |
| Use 20 rather than 0 ViT encoder layers before DeepStack (ablation) | DeepStack-V · ViT evaluation suite | AVG: 41.2 -> 51.3 | Helped |
| Use 16 rather than 20 ViT encoder layers before DeepStack (ablation) | DeepStack-V · ViT evaluation suite | AVG: 51.3 -> 50.9 | Hurt |
| Increase the number of LLM layers used for stacking high-resolution tokens (ablation) | DeepStack · Phi-3 evaluation suite | AVG | Helped |
Surprising findings
- Stacking duplicated original (dummy) visual tokens gives no gain, indicating that the improvement comes from high-resolution tokens rather than the residual connection itself.
- Inserting visual tokens into the early decoder layers (before about the midpoint) keeps performance largely intact, suggesting LLMs can process visual inputs from early layers rather than only the input layer.
Limitations
- The stacking design (start layer, number of layers, residual insertion) is heuristic and not systematically optimized, as the authors acknowledge.
- The model lags behind LLaVA-NeXT on some benchmarks (e.g., MM-Vet, MMMU) because it uses a different SFT mixture.
- Video QA is evaluated only zero-shot via frame mosaics, and gains on video are modest.
Experimental setup
Datasets: LCS-558k LLaVA-mixed-665k custom 748K SFT mixture following LLaVA-NeXT
Benchmarks: VQAv2 GQA TextVQA DocVQA InfoVQA SEED POPE MMMU MM-Vet ChartQA MultiDocVQA EgoSchema
Metrics: accuracy score
Baselines: LLaVA-1.5 LLaVA-NeXT VILA Qwen-VL InstructBLIP BLIP-2
Builds on: LLaVA-1.5 LLaVA-NeXT CLIP
Closest papers by meaning
- iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models (ICLR 2026 · similarity 0.91)
- Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment (AAAI 2026 · similarity 0.90)
- ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models (CVPR 2025 · similarity 0.90)
- LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token (ICLR 2025 · similarity 0.90)
- One Layer's Trash is Another Layer's Treasure: Adaptive Layer-wise Visual Token Selection in LVLMs (CVPR 2026 · similarity 0.90)
- Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.