Research map / Vision-language and multimodal / Multimodal LLMs

VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VISOR, which cuts LVLM inference cost by sparsifying text-image and image-image attention (cross-attention layers plus a few self-attention layers on visual tokens, with a universal model and routing policy) instead of pruning visual tokens.

Problem

Efficient LVLM inference is usually achieved by visual token reduction, which creates an information bottleneck that hurts fine-grained, high-resolution tasks such as DocVQA and ChartQA.

Approach

Keep all high-resolution visual tokens but execute only a few cross-attention layers (text queries visual tokens, visual tokens static) plus a few selected self-attention layers that refine visual tokens; train one universal model across self-attention budgets and route per sample with a lightweight policy trained on offline pseudo-labels.

Key results

• VISOR (LLaVA-OV 0.5B) reaches 63.6 average on easy and 58.4 on hard tasks at 8.6x average FLOP savings, vs 61.5/57.1 for the full LLaVA-OV baseline.
• VISOR-TR reaches 18x FLOP savings with 63.3 easy and 57.8 hard average, while the best token-reduction baseline M^3 gets 64.0/56.6 at 8.0x.
• On DocVQA, VISOR scores 68.9 vs 68.7 for LLaVA-OV, while VisionZip drops to 36.7 at 5.7x savings.

Surprising findings

• Universal training across configurations matched or surpassed independently trained fixed-budget models.
• Dropping all vision tokens from random LLM layer subsets leaves easy tasks robust but causes sharp drops on hard tasks such as DocVQA and ChartQA.

Limitations

• Routing policy and universal training depend on offline pseudo-labels and per-dataset labels, and the router is trained on a fixed set of viable configurations.
• Training data partially unavailable, so the model is trained on a smaller set than LLaVA-OV, and Qwen2-VL comparison is at a disadvantage.
• Gains on some benchmarks are mixed; VISOR trails the baseline on MME and MMStar at 0.5B and on InfoVQA with Qwen2-VL-2B.

Experimental setup

Datasets: CC3M COCO118K BLIP558K SynthDog Evol-Instruct LLaVA-OV Single-Image 3.2M

Benchmarks: RealWorldQA ScienceQA GQA MME MMStar MMBench POPE AI2D ChartQA TextVQA InfoVQA OCRBench

Metrics: accuracy MME score FLOPs savings wall-clock latency

Baselines: VisionZip VisPruner SparseVLM PyramidDrop M^3 HiRED Downsample

Builds on: LLaVA-OneVision VisionZip VisPruner Matryoshka Multimodal Models

Closest papers by meaning

• A-VL: Adaptive Attention for Large Vision-Language Models (AAAI 2025 · similarity 0.91)
• AdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisition (CVPR 2026 · similarity 0.90)
• Attention-aware Inference Optimizations for Large Vision-Language Models with Memory-efficient Decoding (CVPR 2026 · similarity 0.90)
• A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs (CVPR 2025 · similarity 0.90)
• Conical Visual Concentration for Efficient Large Vision-Language Models (CVPR 2025 · similarity 0.90)
• Multi-step Visual Reasoning with Visual Tokens Scaling and Verification (NeurIPS 2025 · similarity 0.89)