Research map / Vision-language and multimodal / Multimodal LLMs

Conical Visual Concentration for Efficient Large Vision-Language Models

CVPR 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PyramidDrop, which splits the LLM into stages and drops image tokens at stage boundaries using last-instruction-token attention, cutting LVLM training time ~40% and inference FLOPs ~55% with comparable performance.

Problem

Large vision-language models spend huge compute on hundreds to thousands of image tokens, and prior token-reduction methods either compress before the LLM or drop tokens at very shallow layers, losing information.

Approach

Visual redundancy grows with depth, so drop image tokens progressively at the end of each of several LLM stages (ratio λ, default 0.5, S=4), ranking tokens by attention from the last instruction token; this is used both for training acceleration and as a training-free inference method.

Key results

• On LLaVA-NeXT-7B, PyramidDrop cut training GPU hours from 366 to 218 (40.4% reduction) with average benchmark score 67.5 vs 67.6 vanilla.
• With 9 patches, PyramidDrop training took 269 vs 483 GPU hours (44.3% reduction) and achieved higher average (67.4 vs 66.8) on 8 benchmarks.
• As inference-only method on LLaVA-NeXT-7B, PyramidDrop reached 70.5 average vs FastV 70.1 and vanilla 70.9 at 9.5 vs 10.6 TFLOPS (FastV) and 20.8 (vanilla).

Measured effects reported in the paper

Surprising findings

• Models trained with PyramidDrop retain more TextVQA performance at a given token retention ratio than vanilla models, suggesting training with progressive pruning encourages denser image representations.
• Video LLM experiments show even aggressive token pruning barely affects zero-shot video QA accuracy, attributed to high inter-frame redundancy.

Limitations

• Gains on video models are smaller and the authors note video redundancy requires specialized designs.
• LLaVA-NeXT training data and code are not open-source, so training was reproduced on Open-LLaVA-NeXT with differing data, so numbers may differ from the original.
• Stage count S and drop ratio λ are fixed by heuristic (S=4, λ=0.5) and only partly ablated.

Experimental setup

Datasets: TextVQA DocVQA InfoVQA ChartQA OCRVQA VQAv2 MME MMBench MMBench-CN SEED-Bench MM-Vet VizWiz

Benchmarks: MME MMBench MMBench-CN SEED-Bench MM-Vet VQAv2 VizWiz DocVQA ChartQA InfoVQA TextVQA MMStar

Metrics: accuracy GPU hours TFLOPs GPT-evaluation score

Baselines: FastV vanilla LLaVA-NeXT-7B vanilla LLaVA-1.5-7B Video-LLaVA

Builds on: FastV FlashAttention

Closest papers by meaning

• A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs (CVPR 2025 · similarity 0.91)
• Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters (ICLR 2025 · similarity 0.91)
• A-VL: Adaptive Attention for Large Vision-Language Models (AAAI 2025 · similarity 0.91)
• Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.91)
• SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference (ICML 2025 · similarity 0.90)
• ViCO: A Training Strategy towards Semantic Aware Dynamic High-Resolution (NeurIPS 2026 · similarity 0.90)