Research map / Vision-language and multimodal / Multimodal LLMs

Run, Ruminate, and Regulate: A Dual-process Thinking System for Vision-and-Language Navigation

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Reinforcement learning for visual reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes R^3, a zero-shot dual-process VLN framework that pairs a lightweight transformer expert (Runner) with GPT-4o-based chain-of-thought reasoning (Ruminator), switched by a Regulator using looping, GNN scoring, and ending criteria.

Problem

Vision-and-language navigation agents lag behind domain-expert models when fully delegated to LLMs, which struggle with real-world spatial layouts, and LLM inference is costly and slow.

Approach

A dual-process system where a fast VLN expert handles routine steps and a multimodal LLM with CoT prompting takes over only when a learned/rule-based regulator detects anomalies (looping, GNN-scored likelihood of failure, or a suspicious STOP), with a restart-formulation stage.

Key results

• On REVERIE val unseen, R^3 reaches 53.76 SR, 42.14 SPL, 37.94 RGS, 29.86 RGSPL, exceeding prior best by 3.28 SPL and 3.30 RGSPL.
• On R2R val unseen, R^3 reports 77 SR and 66 SPL with NE 2.76.
• Removing the scoring criterion drops REVERIE SR by 2.05 and SPL by 2.76 points; the paper reports R^3 inference time about one-fifth of other LLM-assisted methods.

Surprising findings

• Without the Ruminator (Runner alone), results are close to the full system's SR gap suggesting the LLM contributes modest gains; the Runner alone already outperforms MiniGPT-4 as Ruminator backbone.

Limitations

• Relies on proprietary GPT-4o via API for the Ruminator, adding cost and latency dependence.
• Thresholds (tau_r, tau_l, tau_g) are hand-set hyperparameters.
• Reported R2R/REVERIE gains are mainly on the unseen validation split; no test-set results are reported in the text.

Experimental setup

Datasets: R2R REVERIE Matterport3D

Benchmarks: R2R REVERIE

Metrics: TL NE SR SPL RGS RGSPL

Baselines: GridMM DUET BEVBert SUSA NavGPT MapGPT NavGPT-2 NaviLLM MC-GPT GPT-3.5 Turbo MiniGPT-4

Builds on: GridMM NavGPT MapGPT chain-of-thought prompting dual-process theory

Closest papers by meaning

• NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (AAAI 2024 · similarity 0.91)
• Aux-Think: Exploring Reasoning Strategies for Data-Efficient Vision-Language Navigation (NeurIPS 2025 · similarity 0.90)
• FantasyVLN: Unified Multimodal Chain-of-Thought Reasoning for Vision-and-Language Navigation (CVPR 2026 · similarity 0.90)
• D3D-VLP: Dynamic 3D Vision-Language-Planning Model for Embodied Grounding and Navigation (CVPR 2026 · similarity 0.90)
• STRIDER: Navigation via Instruction-Aligned Structural Decision Space Optimization (NeurIPS 2025 · similarity 0.90)
• Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigation (NeurIPS 2025 · similarity 0.89)