Research map / Vision-language and multimodal / Multimodal LLMs

Focus, Don't Prune: Identifying Instruction-Relevant Regions for Information-Rich Image Understanding

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes PinPoint, a two-stage method that selects instruction-relevant regions via learnable guidance queries and contrastive alignment, then re-encodes them for LVLM VQA, plus new region annotations for InfoVQA, SPDocVQA, and MPDocVQA.

Problem

Information-rich images (infographics, document layouts) require many visual tokens at high resolution, making LVLM inference expensive; attention-based token pruning is unreliable and causes hallucination.

Approach

Instead of pruning tokens by attention, a lightweight module with learnable guidance queries aligns sliding-window region features with the instruction (inter-modal and intra-image contrastive losses), selects top regions up to a coverage ratio, and re-encodes only those regions for the LLM.

Key results

• On LLaVA-NeXT-7B, PinPoint reaches 0.3024 ANLS on InfoVQA vs 0.2552 for vanilla, using 65.3% of FLOPs.
• On Qwen2-VL-7B, PinPoint achieves 0.7140 ANLS on InfoVQA at 55.5% of vanilla FLOPs (vanilla 0.7399).
• Trainable parameters are 99.4M (1.40%) for LLaVA-NeXT-7B and 72.6M (0.86%) for Qwen2-VL-7B; training takes 1.33 GPU hours on two A100s.

Surprising findings

• Attention-based pruning methods FastV and PyramidDrop degrade substantially on Qwen2-VL, with InfoVQA ANLS dropping from 0.7399 to about 0.45-0.51.
• PinPoint trained only on GQA improves TextVQA by about 36.9% relative to vanilla (0.7293 vs 0.5327).

Limitations

• Pipeline for dataset annotation relies on proprietary models (GPT-4o) and OCR (Amazon Textract), and some samples needed manual correction.
• Gains on Qwen2-VL with region-only input are not uniform; GQA and some datasets show near-parity with vanilla.
• Coverage threshold r must be set manually and affects performance differently across datasets.

Experimental setup

Datasets: InfoVQA SPDocVQA MPDocVQA GQA

Benchmarks: InfoVQA SPDocVQA MPDocVQA GQA MMMU MMMU-Pro TextVQA POPE CHAIR AMBER

Metrics: ANLS accuracy FLOPs latency region accuracy region coverage

Baselines: ToMe FastV PyramidDrop SparseVLM ViCrop Random sampling Honeybee

Builds on: ViCrop FastV PyramidDrop LLaVA-NeXT

Closest papers by meaning

• LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.89)
• VIGC: Visual Instruction Generation and Correction (AAAI 2024 · similarity 0.89)
• The Narrow Gate: Localized Image-Text Communication in Native Multimodal Models (NeurIPS 2025 · similarity 0.89)
• Osprey: Pixel Understanding with Visual Instruction Tuning (CVPR 2024 · similarity 0.88)
• VLM-R³: Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thought (NeurIPS 2025 · similarity 0.88)
• Multi-step Visual Reasoning with Visual Tokens Scaling and Verification (NeurIPS 2025 · similarity 0.88)