Research map / Vision-language and multimodal / Multimodal LLMs

Seeing What Matters: A Training-Free Self-Guided Framework for Multimodal Detail Perception and Reasoning

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Instruction-tuned multimodal assistants

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes SLoFo, a training-free self-guided inference framework that uses gradient-weighted attention from a planning anchor plus PCA reconstruction error to crop a detail sub-image, and phase-wise visual token pruning to improve fine-grained perception in LLaVA-v1.5-7B.

Problem

Fixed-resolution MLLMs struggle to perceive fine-grained visual details, suffering from distracted attention (attention sinks on irrelevant regions) and blurry vision (limited input resolution).

Approach

Combines a gradient-weighted semantic relevance map (gradient of a pseudo-token w.r.t. attention from a planning anchor) with a structure-based PCA reconstruction-error map to crop a question-relevant sub-image, then progressively prunes original-image visual tokens across LLM phases during inference.

Key results

• SLoFo reaches 63.02% on TextVQA (+4.79 over LLaVA-v1.5-7B) and 27.69% on DocVQA (+12.01).
• SLoFo high-res achieves 61.78% on V* and 46.125% on HR-Bench-4K, outperforming training-free baselines including DyFo-LLaVA on V*.
• On POPE adversarial MSCOCO, SLoFo high-res improves F1 by about 4.75 points over the LLaVA-v1.5-7B baseline; GQA improves by 2.58% in high-res setting.

Surprising findings

• Pure attention-based selection gives limited gains versus gradient-weighted attention, attributed to attention sinks.
• The default SLoFo slightly underperforms DyFo-LLaVA on V* (57.59% vs 59.16%).

Limitations

• Sub-image cropping adds visual tokens and computational cost.
• Single sub-image design limits performance on relational reasoning and counting.
• Evaluated primarily on VQA-style tasks, not open-ended dialogue or multi-turn reasoning.

Experimental setup

Datasets: TextVQA POPE DocVQA V* HR-Bench-4K HR-Bench-8K GQA SEED-Bench VQAv2 ScienceQA VizWiz MME

Benchmarks: TextVQA POPE DocVQA V* HR-Bench-4K HR-Bench-8K GQA SEED-Bench VQAv2 ScienceQA VizWiz MME

Metrics: accuracy precision recall F1 MME score

Baselines: InstructBLIP-Vicuna-7B MiniGPT4-Vicuna-7B DeepSeek-VL-7B LLaVA-v1.5-7B ViCrop-LLaVA DyFo-LLaVA SEAL QwenVL-chat

Builds on: ViCrop DyFo SEAL LLaVA-v1.5

Closest papers by meaning

• Zooming without Zooming: Region-to-Image Distillation for Fine-Grained Multimodal Perception (ICML 2026 · similarity 0.91)
• Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.90)
• BVS: Bayesian Visual Search with Multimodal Large Language Model for Fine-grained Perception (ICML 2026 · similarity 0.90)
• Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation (NeurIPS 2026 · similarity 0.90)
• Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs (ICML 2026 · similarity 0.90)
• SIFThinker: Spatially-Aware Image Focus for Visual Reasoning (AAAI 2026 · similarity 0.90)