Research map / Vision-language and multimodal / Visual question answering
FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FOCUS, a training-free visual cropping method that builds an object relevance map from MLLM KV-cache value-feature cosine similarity to localize and crop regions for fine-grained VQA.
Problem
MLLMs struggle with small visual details in high-resolution images for VQA; existing visual cropping methods need task-specific fine-tuning, exhaustive search, or are incompatible with efficient attention such as FlashAttention.
Approach
Localize question-relevant objects by computing cosine similarity between cached value features of target-object text tokens and visual tokens (a value-value pseudo-attention aggregated over layers via attention rollout), then rank proposed ROIs with an MLLM existence confidence; this is compatible with FlashAttention and needs only one forward pass for the search.
Key results
- FOCUS reaches 72.77% on V*Bench, 51.75% on HRBench-4K and 45.00% on HRBench-8K with LLaVA-1.5 using fewer than 17 forward passes, while ZoomEye reaches 77.48% on V*Bench at 3.43x higher cost.
- With LLaVA-OneVision, FOCUS raises V*Bench (2K) accuracy from 74.46% to 92.15% and is on average 5.47x more efficient than ZoomEye on MME-RealWorld-Lite.
- FOCUS is reported to match ZoomEye's accuracy while requiring 3-6.5x less compute across the evaluated settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FOCUS visual cropping instead of vanilla inference | LLaVA-OneVision-7B · V*Bench | accuracy: 74.46% to 92.15% | Helped |
| Use FOCUS instead of ZoomEye | LLaVA-1.5-7B · HRBench-4K | accuracy: 50.00 to 51.75% | Helped |
| Use FOCUS instead of ZoomEye | LLaVA-1.5-7B · HRBench-8K | accuracy: 45.00% vs 49.00% | Hurt |
| Use FOCUS instead of ZoomEye | LLaVA-OneVision-7B · MME-RealWorld-Lite Perception | average accuracy: 56.29 vs 54.15 | Hurt |
| Use FOCUS instead of ZoomEye | LLaVA-OneVision-7B · MME-RealWorld-Lite Reasoning | average accuracy: 43.20 vs 44.53 | Helped |
| Replace the object relevance map with a randomly generated map (ablation) | FOCUS · V*Bench and HRBench-4K | accuracy | Hurt |
| Discard ROI ranking and select the ROI with the highest object relevance score (ablation) | FOCUS · V*Bench | recall: 38.48% | Hurt |
| Discard ROI ranking and select the ROI with the highest object relevance score (ablation) | FOCUS · V*Bench and HRBench-4K | accuracy: +3 percentage points on V*Bench; +5 pp. on HRBench-4K | Helped |
| Add ROI ranking based on existence confidence (ablation) | FOCUS · V*Bench and HRBench-4K | accuracy: -3 pp. on V*Bench; -5 pp. on HRBench-4K versus the no-ranking variant | Hurt |
| Replace value features with key features, removing RoPE from keys (ablation) | FOCUS · V*Bench and HRBench-4K | accuracy | Hurt |
| Use later layers 14-32 rather than earlier layers 0-14 or all layers 0-32 (ablation) | FOCUS · V*Bench and HRBench-4K | accuracy | Helped |
| Increase the number of ROI-ranking steps (ablation) | FOCUS · V*Bench | recall | Helped |
Surprising findings
- Using a randomly generated relevance map still yields accuracy well above random guessing (35.99%), indicating ROI confidence-based selection is robust.
- Key features with RoPE removed performed worse than value features for building the relevance map.
Limitations
- Relevance map resolution limits localization of tiny objects, hurting performance on HRBench-8K with LLaVA-1.5.
- Inherits weak spatial-relationship understanding from base MLLM, struggling with left/right concepts.
- Baseline ZoomEye and SEAL evaluated under a different inference scheme; DC2 results taken from original paper with estimated FPs.
Experimental setup
Benchmarks: V*Bench HRBench-4K HRBench-8K MME-RealWorld-Lite A-OKVQA GQA
Metrics: accuracy forward passes (FP) recall execution time peak GPU memory
Baselines: SEAL DC2 ZoomEye ViCrop LLaVA-1.5-7B LLaVA-OneVision-7B
Builds on: ViCrop ZoomEye attention rollout FlashAttention Logit Lens
Closest papers by meaning
- Multi-Level Attention Networks for Visual Question Answering (CVPR 2017 · similarity 0.92)
- Where to Look: Focus Regions for Visual Question Answering (CVPR 2016 · similarity 0.92)
- MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMs (ICLR 2025 · similarity 0.90)
- Rethinking Token Reduction for Large Vision-Language Models (CVPR 2026 · similarity 0.90)
- BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions (AAAI 2024 · similarity 0.90)
- When Open-Vocabulary Visual Question Answering Meets Causal Adapter: Benchmark and Approach (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.