Research map / Vision-language and multimodal / Multimodal LLMs

Retrieval-Augmented Perception: High-resolution Image Perception Meets Visual RAG

ICML 2025 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Retrieval-Augmented Perception (RAP), a training-free framework that retrieves query-relevant image crops with VisRAG, re-lays them out preserving spatial positions, and selects the number of crops via an A*-style RE-Search guided by similarity and MLLM confidence.

Problem

MLLMs process images at fixed low resolution, causing distortion and detail loss on high-resolution images; existing crop, HR-encoder, and search methods either downsample or search top-down with unreliable early-stage perception.

Approach

Treat HR perception as a long-context RAG problem: retrieve query-relevant image crops with a visual retriever, reassemble them into a compact image while keeping their relative spatial layout (Spatial-Awareness Layout), and adaptively choose the number of crops with an A*-like search (RE-Search) that combines retrieval similarity and the model's Yes-confidence.

Key results

• LLaVA-v1.5-13B with RAP reaches 89.8% overall on V* Bench (vs 47.1% baseline) and 59.3% on HR-Bench 8K (vs 37.8%).
• RAP with LLaVA-v1.5-7B improves HR-Bench 8K overall from 32.1% to 53.8% (+21.7), outperforming Zoom Eye (50.8%) and DC^2 (34.9%).
• RAP achieves 4.2 samples/min throughput at 60.1% HR-Bench 4K accuracy with LLaVA-v1.5-13B, versus Zoom Eye's 3.3 and 58.0%.

Measured effects reported in the paper

Surprising findings

• Random crop retention at K=4 or half the crops performs far worse than query-based retrieval, and worse than the baseline on HR-Bench 8K.
• Retaining more crops helps cross-instance tasks but too many crops hurts single-instance tasks due to higher resolution.

Limitations

• Performance depends on retriever quality; a weaker SigLIP retriever gives smaller gains than VisRAG.
• Gains on diagram and table tasks in MME-RealWorld are limited, attributed to weak spatial reasoning in MLLMs.
• Search requires repeated MLLM confidence queries and up to 200 steps, adding inference overhead.

Experimental setup

Datasets: HR-Bench V* Bench MME-RealWorld DocVQA

Benchmarks: V* Bench HR-Bench 4K HR-Bench 8K MME-RealWorld

Metrics: accuracy throughput (samples per minute)

Baselines: DC^2 Zoom Eye LLaVA-v1.6-7B LLaVA-v1.6-13B LLaVA-HR InternVL-1.5-26B GPT-4o Qwen-VL-max

Builds on: VisRAG A* search DC^2 Zoom Eye

Closest papers by meaning

• CVSearch: Empowering Multimodal LLMs with Cognitive Visual Search for High-Resolution Image Perception (ICML 2026 · similarity 0.89)
• MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models (ICLR 2025 · similarity 0.88)
• Divide, Conquer and Combine: A Training-Free Framework for High-Resolution Image Perception in Multimodal Large Language Models (AAAI 2025 · similarity 0.88)
• AR-RAG: Autoregressive Retrieval Augmentation for Image Generation (NeurIPS 2025 · similarity 0.88)
• GenIR: Generative Visual Feedback for Mental Image Retrieval (NeurIPS 2025 · similarity 0.88)
• Fix Before Search: Benchmarking Agentic Visual Query Pre-processing in Multimodal Retrieval-augmented Generation (ICML 2026 · similarity 0.87)