Research map / Vision-language and multimodal / Multimodal LLMs

MMhops-R1: Multimodal Multi-hop Reasoning

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces MMhops, a 31K-sample multimodal multi-hop VQA benchmark (Bridging and Comparison), and MMhops-R1, an RL-trained (GRPO/DAPO-style) mRAG agent on Qwen2.5-VL-7B that interleaves image and text retrieval.

Problem

Existing knowledge-based VQA benchmarks and mRAG systems handle only single-step visual recognition plus one retrieval, lacking multi-hop reasoning that spans visual and textual modalities with dynamic retrieval.

Approach

Builds multi-hop questions by iteratively chaining Wikipedia entities (bridging) and cross-image comparisons, then trains a Qwen2.5-VL agent with RL to choose among image search, text search, and answer actions, using a reward gated by outcome and format.

Key results

• MMhops-R1 (Qwen2.5-VL-7B) reaches 51.35% overall on MMhops Bridging and 22.01% on Comparison, beating OmniSearch (GPT-4o) at 42.65% and 17.02%.
• On INFOSEEK, MMhops-R1 achieves 33.2% overall, above EchoSight at 31.3%.
• On E-VQA two-hop, MMhops-R1 scores 23.3, above PaLM at 22.8.

Surprising findings

• Self-Ask with GPT-4o drops below plain GPT-4o on comparison questions (30.42% vs 36.62% on bridging; 18.27 vs 8.76 comparison shows it is higher, so the drop claim is mixed in the text).
• Gemini-2.5-pro remains the top performer, ahead of the proposed 7B model on MMhops.

Limitations

• Retrieval knowledge base is limited to 100K Wikipedia articles with images, a restricted setting.
• Performance on comparison questions remains low (22.01%) and below Gemini-2.5-pro.
• Dataset construction relies on GPT-4o generation and LLM-based quality control, which may introduce errors.

Experimental setup

Datasets: MMhops INFOSEEK E-VQA OK-VQA A-OKVQA ViQuAE

Benchmarks: MMhops INFOSEEK E-VQA

Metrics: exact match accuracy with tolerance for time and numerical answers IoU for numerical answers overall weighted accuracy

Baselines: GPT-4o-mini GPT-4o Gemini-2.5-flash Gemini-2.5-pro Qwen2.5-VL-7B-Instruct Qwen2.5-VL-72B-Instruct Search-R1 Self-Ask Vanilla mRAG EchoSight OmniSearch CLIP-PaLM

Builds on: Search-R1 DAPO GRPO INFOSEEK E-VQA

Closest papers by meaning

• MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chains (ICLR 2026 · similarity 0.88)
• MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.88)
• Towards Robust Multi-Modal Reasoning via Model Selection (ICLR 2024 · similarity 0.88)
• R-4B: Incentivizing General-Purpose Auto-Thinking in MLLMs via Bi-Mode Annealing and Reinforce Learning (CVPR 2026 · similarity 0.88)
• MMRAG-RFT: Two-stage Reinforcement Fine-tuning for Explainable Multi-modal Retrieval-augmented Generation (AAAI 2026 · similarity 0.87)
• MM-HELIX: Boosting Multimodal Long-Chain Reflective Reasoning with Holistic Platform and Adaptive Hybrid Policy Optimization (ICLR 2026 · similarity 0.87)