Research map / Vision-language and multimodal / Multimodal LLMs

ReFAct: Empowering Multimodal Web Agents with Visual and Context Focusing

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Multimodal LLM benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ReFAct, a multimodal web search agent with active visual grounding (bounding-box crops) and memory Defocus/Refocus operations, trained with GRPO on a new GroundedVQA dataset to handle visual and retrieval noise.

Problem

Multimodal web search agents are easily misled by visual noise (irrelevant background, cluttered scenes) and retrieval noise (ads, navigation bars), which corrupts search queries and reasoning.

Approach

Let the agent actively manage its cross-modal context: predict bounding boxes to crop image regions before image search (Grounding), and offload/restore long retrieval content to an external memory via Defocus/Refocus, all trained end-to-end with RL.

Key results

• ReFAct-7B reaches 0.513 on GroundedVQA Level-1 and 0.375 on Level-2, above MMSearch-R1 (0.433 / 0.304) and WebWatcher (0.372 / 0.232).
• Removing visual focusing (Grounding tool) drops ReFAct-7B Level-1 accuracy from 51.3 to 40.2 (-11.1).
• On the extreme-noise bin (target ratio <5%), ReFAct-7B gains +6.3 points over MMSearch-R1.

Surprising findings

• The ReFAct plug-in framework gives negligible or negative gains for closed-source models (Gemini-2.5-flash, GPT-5-mini), attributed to weaker innate grounding.
• ReFAct-7B trails Gemini-2.5-flash and GPT-5-mini on general benchmarks SimpleVQA and LiveVQA.

Limitations

• RL trained only on visual grounding; Defocus/Refocus memory tools not trained because RL with dynamic context was judged unstable.
• Weaker on general benchmarks (SimpleVQA, LiveVQA) where region-level focus reduces search query richness.
• Benchmark is small (261 Level-1 and 56 Level-2 test items) and relies on LLM-as-judge.

Experimental setup

Datasets: GroundedVQA FVQA SA-1B

Benchmarks: GroundedVQA MMSearch SimpleVQA LiveVQA

Metrics: pass@1 accuracy

Baselines: Qwen2.5-VL-7B Qwen2.5-VL-32B Qwen2.5-VL-72B GPT-5-mini Gemini-2.5-flash DeepEyes WebWatcher MMSearch-R1

Builds on: GRPO ReAct Search-R1 MMSearch-R1 WebWatcher

Closest papers by meaning

• Multimodal Web Navigation with Instruction-Finetuned Foundation Models (ICLR 2024 · similarity 0.89)
• OpenSearch-VL: An Open Recipe for Frontier Multimodal Search Agents (NeurIPS 2026 · similarity 0.89)
• MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents (ICLR 2026 · similarity 0.89)
• GPT-4V(ision) is a Generalist Web Agent, if Grounded (ICML 2024 · similarity 0.89)
• VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Tasks (ICLR 2025 · similarity 0.88)
• MM-DeepResearch: A Simple and Effective Multimodal Agentic Search Baseline (ICML 2026 · similarity 0.87)