Research map / Vision-language and multimodal / Multimodal LLMs
HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HiRED, a training-free, plug-and-play visual token dropping method for high-resolution VLMs that allocates a fixed token budget across image partitions using ViT CLS attention and keeps the most informative tokens before the LLM.
Problem
High-resolution VLMs with dynamic partitioning generate many visual tokens, causing high latency, low throughput and GPU memory pressure on resource-constrained commodity GPUs.
Approach
Uses ViT CLS-attention from an initial layer on the full image to allocate per-partition token budgets, and CLS-attention from the final layer to select the top-scoring tokens within each partition's budget, dropping tokens early before the LLM.
Key results
- HiRED-20% on LLaVA-Next-7B gives 4.7x token generation throughput (2.30 vs 0.49 tokens/sec) and 78% lower latency (4.21 vs 19.49 s) on a 24GB P40.
- At batch size 4, full execution runs out of memory on the P40 while HiRED-20% uses 16.99 GB, a 30% memory reduction.
- HiRED-20% trails PruMerge+ (55% tokens) by 11% on TextVQA and 26% on DocVQA, showing better accuracy than early-dropping baselines; HiRED-20% maintains near-full accuracy on VQA tasks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| HiRED token dropping at 20% budget vs full tokens | LLaVA-Next-7B · TextVQA / DocVQA / ChartQA | accuracy: TextVQA 64.8→61.4; DocVQA 73.4→60.8; ChartQA 54.8→42.0 | Hurt |
| HiRED token budget raised from 20% to 40% | LLaVA-Next-7B · DocVQA / ChartQA | accuracy: DocVQA 60.8→68.7; ChartQA 42.0→46.5 | Helped |
| HiRED dropping at 20% budget vs full tokens | LLaVA-Next-7B · VQA-v2 / ScienceQA | accuracy: VQA-v2 77.5 vs 80.3; ScienceQA 73.4 vs 73.2 | No clear effect |
| HiRED at 20% and 40% budget vs PruMerge+ on transcription tasks | LLaVA-Next-7B · TextVQA / DocVQA | accuracy: HiRED-20% 61.4 vs PruMerge+ 54.4 on TextVQA; 60.8 vs 44.6 on DocVQA | Helped |
| CLS-attention-based budget allocation (layer 0) vs even budget split (ablation) | HiRED on LLaVA-Next-7B · TextVQA / DocVQA | accuracy: TextVQA 52.2→54.8; DocVQA 48.8→59.2 | Helped |
| Dropping tokens by final layer (22) CLS attention vs layer 11 (ablation) | HiRED on LLaVA-Next-7B · TextVQA / DocVQA | accuracy: TextVQA 51.9→54.8; DocVQA 52.0→59.2 | Helped |
| Using final-layer CLS attention for budget allocation (layer 22) instead of layer 0 (ablation) | HiRED on LLaVA-Next-7B · TextVQA / DocVQA | accuracy: TextVQA 54.8→37.3; DocVQA 59.2→54.0 | Hurt |
| Summing CLS attention across heads vs no head aggregation (ablation) | HiRED on LLaVA-Next-7B · TextVQA / DocVQA | accuracy: TextVQA 50.5→54.8; DocVQA 52.4→59.2 | Helped |
| HiRED token dropping at 20% budget on LLaVA-Next-7B vs full execution | LLaVA-Next-7B · BS1 inference efficiency | throughput / latency / memory: 0.49→2.30 tokens/s; 19.49→4.21 s; 16.04→13.76 GB | Helped |
| HiRED 20% budget at batch size 4 | LLaVA-Next-7B · BS4 memory | GPU memory (GB): Full OOM → 16.99 GB | Helped |
| HiRED token dropping on low-resolution LLaVA-1.5-7B | LLaVA-1.5-7B · VQA-v2 / SQA / TextVQA / DocVQA | accuracy: VQA-v2 40%: 79.0 vs 76.6 full; 20%: 74.7; SQA 67.2 vs 69.5 at 40% | Mixed |
Surprising findings
- Accuracy on ScienceQA and POPE increases with fewer visual tokens in some settings.
- Using CLS-attention from the initial ViT layer to allocate budget works better than deeper layers, while dropping based on the final layer works best.
Limitations
- Loss of spatial information from dropped tokens hurts spatial tasks such as ChartQA.
- Accuracy drops noticeably on fine-grained transcription tasks at low budgets (e.g., DocVQA at 20%).
- Evaluated only on LLaVA-family and ShareGPT4V models with CLIP ViT encoders.
Experimental setup
Datasets: VQA-v2 ScienceQA TextVQA DocVQA OCRBench MME POPE ChartQA
Benchmarks: VQA-v2 ScienceQA TextVQA DocVQA OCRBench MME POPE ChartQA
Metrics: accuracy token generation throughput (tokens/sec) time-to-first-token latency (sec) GPU memory (GB)
Baselines: PruMerge PruMerge+ Spatial pooling FastV FlexAttention TokenCorrCompressor
Builds on: PruMerge LLaVA-Next FastV
Closest papers by meaning
- Variation-aware Vision Token Dropping for Faster Large Vision-Language Models (CVPR 2026 · similarity 0.92)
- Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters (ICLR 2025 · similarity 0.92)
- iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models (ICLR 2026 · similarity 0.90)
- DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference (CVPR 2026 · similarity 0.90)
- ApET: Approximation-Error Guided Token Compression for Efficient VLMs (CVPR 2026 · similarity 0.90)
- Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment (AAAI 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.