Research map / Vision-language and multimodal / Multimodal LLMs

FantasyVLN: Unified Multimodal Chain-of-Thought Reasoning for Vision-and-Language Navigation

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes FantasyVLN, a unified implicit multi-CoT framework that compresses imagined visual observations into VAR latent tokens, jointly trains non-CoT, textual, visual and multimodal CoT modes with cross-mode alignment, and infers without generating CoT, evaluated on LH-VLN.

Problem

Multimodal chain-of-thought for vision-and-language navigation causes severe token inflation (3k-5k tokens per reasoning step), making real-time navigation impractical, while textual CoT lacks spatial grounding and overfits to sparse annotations.

Approach

Train one Qwen2.5-VL policy with gating tokens to switch among non-CoT, textual, latent-visual (VAR-compressed) and multimodal CoT modes, align all CoT action predictions to the non-CoT prediction via a soft-target alignment loss, and at inference use only direct instruction-to-action decoding.

Key results

• On LH-VLN online test (unseen tasks and scenes), FantasyVLN reaches SR 2.44, ISR 11.01, CSR 9.64, CGT 8.99, the best among compared methods including Aux-Think (SR 0.65, ISR 3.16).
• Implicit inference reaches 1.03 actions per second versus 0.19 for explicit CoT-VLA (7B), supporting the claimed real-time benefit.
• Removing the cross-mode alignment constraint drops SR from 2.44 to 0 and ISR from 11.01 to 2.39.

Surprising findings

• Explicit CoT decoding underperforms implicit inference for every CoT modality (e.g., MM-CoT SR 0.98 explicit vs 2.44 implicit).
• Absolute success rates are very low (SR at most 2.44), indicating the multi-stage LH-VLN task remains largely unsolved.

Limitations

• Absolute navigation success rates are very low (SR 2.44 at best), so gains are small in absolute terms.
• Baselines CoT-VLA and Aux-Think were re-implemented from paper descriptions because training code is not public; WorldVLA was adapted by the authors.
• Only evaluated on LH-VLN; no other VLN benchmark is reported.

Experimental setup

Datasets: LH-VLN

Benchmarks: LH-VLN

Metrics: SR ISR CSR CGT APS

Baselines: Random GLM-4v prompt GPT-4 + NaviLLM MGDM CoT-VLA WorldVLA Aux-Think

Builds on: Aux-Think NavCoT VAR (Visual AutoRegressor) Qwen2.5-VL CoT-VLA

Closest papers by meaning

• Aux-Think: Exploring Reasoning Strategies for Data-Efficient Vision-Language Navigation (NeurIPS 2025 · similarity 0.93)
• NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (AAAI 2024 · similarity 0.92)
• Zebra-CoT: A Dataset for Interleaved Vision-Language Reasoning (ICLR 2026 · similarity 0.91)
• Visual Planning: Let's Think Only with Images (ICLR 2026 · similarity 0.91)
• Uni-CoT: Towards Unified Chain-of-Thought Reasoning Across Text and Vision (ICLR 2026 · similarity 0.91)
• TVI-CoT: Text-Visual Interleaved Chain-of-Thought Reasoning for Multimodal Understanding (ICML 2026 · similarity 0.90)