Research map / Vision-language and multimodal / Multimodal LLMs

VGR: Visual Grounded Reasoning

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes VGR, a LLaVA-NeXT-based reasoning MLLM that detects key regions during chain-of-thought and replays their high-resolution visual memory tokens into the reasoning, trained on a 158K visual-grounded reasoning SFT dataset (VGR-SFT).

Problem

Multimodal chain-of-thought reasoning relies on language-only reasoning, which suffers from language bias and is largely limited to math/science, failing on perception-heavy visual tasks that need fine image details.

Approach

The model emits bounding-box grounding tokens during reasoning, and the corresponding visual tokens are retrieved from a pre-computed high-resolution visual memory pool and inserted into the sequence (replay), trained with SFT plus an auxiliary L1+GIoU box detection loss.

Key results

• VGR-7B with 864 visual tokens reaches 41.7 on MMStar vs 37.2 for the reproduced LLaVA-NeXT-7B, 67.7 vs 58.7 on ChartQA, and 73.7 vs 70.2 on DocVQA.
• With 3024 visual tokens VGR-7B reaches 72.8 on ChartQA and 43.6 on MMStar, while the original LLaVA-NeXT-7B baseline uses 2880 tokens.
• ZoomEye comparison: VGR-7B averages 7.5s per question on V* Bench versus 48.5s for LLaVA-NeXT-7B with ZoomEye, with V* Bench score 67.7 vs 71.7.

Surprising findings

• Direct use of public CoT datasets (LLaVA-CoT, MMPR) under the same setting performs worse than the baseline on several benchmarks such as DocVQA and ChartQA.
• Feeding random box-replay image tokens with a correct box text token only slightly changes results in the appendix table, which conflicts with the claim that accurate replay is crucial.

Limitations

• Implemented only on the LLaVA architecture with Vicuna-7B/13B and CLIP encoder; stronger encoders and LLMs not explored.
• Reinforcement learning not used; authors list RL integration as future work.
• Inference is slower than the non-reasoning baseline due to longer outputs and replay steps; about 2-2.4 replays per question on average.

Experimental setup

Datasets: VGR-SFT LLaVA-558K LLaVA-NeXT-770K Open-R1 distilled dataset AI2D LLaVA-COCO GQA ChartQA DVQA DocVQA OCRVQA

Benchmarks: MMStar ChartQA DocVQA TextVQA InfoVQA AI2D RealWorldQA POPE V* Bench HR-Bench 8K RefCOCO RefCOCOg

Metrics: accuracy ANLS IoU acc@0.5 wall-clock time

Baselines: LLaVA-NeXT-7B Qwen-VL-Chat-7B Visual CoT DeepSeek-VL-7B LLaVA-v1.5-7B LLaVA-CoT MMPR ZoomEye

Builds on: LLaVA-NeXT Visual CoT Chain-of-Spot ZoomEye

Closest papers by meaning

• MMFineReason: Closing the Multimodal Reasoning Gap via Open Data-Centric Methods (NeurIPS 2026 · similarity 0.93)
• VLM-R³: Region Recognition, Reasoning, and Refinement for Enhanced Multimodal Chain-of-Thought (NeurIPS 2025 · similarity 0.93)
• Multi-step Visual Reasoning with Visual Tokens Scaling and Verification (NeurIPS 2025 · similarity 0.92)
• SLVR: Structured Latent Visual Reasoning via Human-like Reasoning Flows (NeurIPS 2026 · similarity 0.92)
• AStar: Boosting Multimodal Reasoning with Automated Structured Thinking (AAAI 2026 · similarity 0.92)
• TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding (ICML 2026 · similarity 0.92)