Research map / Vision-language and multimodal / Multimodal LLMs

Fix Before Search: Benchmarking Agentic Visual Query Pre-processing in Multimodal Retrieval-augmented Generation

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Interleaved image-text generation with MLLMs

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces V-QPP-Bench, a benchmark of 46,700 imperfect visual queries where MLLM agents must diagnose and repair image imperfections with perceptual tools before multimodal RAG retrieval, and shows SFT on a 4B model matches larger proprietary models.

Problem

Multimodal RAG pipelines treat visual queries as clean and static, so geometric distortions, quality degradation, and semantic clutter in real-world images cause retrieval failures.

Approach

Formulates visual query pre-processing as an agentic tool-use task: MLLMs diagnose imperfections, select perceptual tools (rotate, flip, lum, deblur, denoise, locate, crop, fill) with parameters, and refine the query before retrieval; benchmark built by reverse-engineering corruptions so oracle tool traces are known.

Key results

• Semantic ambiguities cause severe retrieval drops, e.g., RealWorld -98.6% and Expand -80.6% Recall@5 relative to original queries.
• Oracle tool restoration recovers near-perfect performance for severe imperfections such as RealWorld, Rotation and Flip, while off-the-shelf MLLMs show limited gains due to poor tool selection and parameter prediction.
• SFT on 1,000 imperfect queries lets Qwen3-VL-4B-Instruct match or surpass GPT-4o in tool selection accuracy and parameter prediction on most imperfection types.

Surprising findings

• Watermark insertion barely changes retrieval recall yet significantly degrades answer quality for some models.
• Oracle preprocessing gives limited or negligible gains for quality degradations (brightness, blur, noise, crop) because of minimal initial impact or irreversible information loss.

Limitations

• Evaluates only single-corruption scenarios, not concurrent real-world corruptions.
• Corruptions are synthetic and reverse-engineered, with oracle tools tailored to them; only two source datasets used.
• Retrieval corpus is a filtered subset rather than full Wikipedia.

Experimental setup

Datasets: InfoSeek ViQuAE Wikipedia image-text pairs

Benchmarks: V-QPP-Bench

Metrics: Recall@5 Substring Exact Match accuracy Tool Selection Accuracy Parameter Score

Baselines: GPT-4o Gemini-2.5-Flash-Lite Qwen3-VL-4B-Instruct DeepSeek-VL2-Small Phi-4-Multimodal Llama-3.2-11B-Vision Mistral-Small-3.1-24B LLaVA-NeXT-8B Qwen3-VL-30B-A3B

Builds on: InfoSeek ViQuAE nomic-embed-text-v1.5 Grounding DINO

Closest papers by meaning

• MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models (ICLR 2025 · similarity 0.92)
• Navigating Massive Visual Context in Retrieval-Augmented Generation via Multimodal Memory Graph (ICML 2026 · similarity 0.89)
• Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent (ICLR 2025 · similarity 0.89)
• VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
• ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledge (NeurIPS 2026 · similarity 0.89)
• MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents (ICLR 2026 · similarity 0.89)