Research map / Vision-language and multimodal / Multimodal LLMs

Magnifying What Matters: Attention-Guided Adaptive Rendering for Visual Text Comprehension

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows VLMs localize evidence in middle-to-late attention layers without using it reliably in visual text comprehension, and proposes AGAR, a training-free method that re-renders pages with attention-selected evidence spans enlarged.

Problem

Visual text comprehension (VTC) renders long text into images for VLMs, but rendering is treated as fixed and VLMs' internal processing of visualized text is poorly understood; failures on evidence-based QA persist even when evidence is localized.

Approach

Use the VLM's own middle-to-late layer attention to pick top-K visual patches, map them to word spans, and re-render the page with those spans enlarged before a second inference pass; no weights or prompt change.

Key results

• On Qwen3-VL-8B short-form HotpotQA, AGAR raises F1 from 39.7 to 46.9 (+18.1%) over plain VQA, outperforming all training-free baselines.
• On the oracle-span preliminary study with Qwen3-VL-8B on HotpotQA, enlarging ground-truth evidence recovers 21.9% of incorrect samples while breaking only 5.5% of correct ones.
• AGAR improves all 12 subtasks on Qwen3-VL-8B (e.g., +28.5% F1 on LB-MSQ, +38.8% on LoCoMo OpenDomain) while keeping visual-token cost roughly 3x compression.

Surprising findings

• Naive random and uniform font enlargement give little or no gain, and prior VQA attention-emphasis methods (VEA, CGR, VAR, AGLA) often hurt performance on VTC, since VTC needs the surrounding text readable.
• Incorrectly answered samples attend to evidence nearly as strongly as correct ones (localization-without-utilization).

Limitations

• Requires access to backbone attention scores, so it does not apply to closed-source VLMs.
• Deeper mechanistic analysis such as circuit-level accounts is left to future work.
• Requires two forward passes per question, doubling inference cost.

Experimental setup

Datasets: NQ HotpotQA TriviaQA NewsQA LongBench LoCoMo

Benchmarks: NQ HotpotQA TriviaQA NewsQA LongBench HotpotQA LongBench 2WikiMultihopQA LongBench MuSiQue LongBench Qasper LoCoMo

Metrics: EM F1

Baselines: Plain VQA Random patch magnification Uniform font scaling VEA-Br VEA-Con CGR VAR AGLA

Builds on: Visual Evidence Augmentation AGLA VAR Glyph

Closest papers by meaning

• LensVLM: Selective Context Expansion for Compressed Visual Representation of Text (NeurIPS 2026 · similarity 0.89)
• Cross-Lingual Text-Rich Visual Comprehension: An Information Theory Perspective (AAAI 2025 · similarity 0.88)
• Marten: Visual Question Answering with Mask Generation for Multi-modal Document Understanding (CVPR 2025 · similarity 0.88)
• Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs (ICLR 2026 · similarity 0.88)
• Learning Interleaved Image-Text Comprehension in Vision-Language Large Models (ICLR 2025 · similarity 0.88)
• WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation (CVPR 2026 · similarity 0.88)