Research map / Vision-language and multimodal / Robot action and navigation

Vision-and-Language Navigation via Causal Learning

CVPR 2024 · Vision-language and multimodal › Robot action and navigation · approach: Pretraining and data augmentation for VLN · 40 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GOAT, a generalized cross-modal causal transformer for vision-and-language navigation that applies back-door and front-door adjustments (BACL, FACL) plus a contrastively trained cross-modal feature pooling module to mitigate observable and unobservable dataset confounders.

Problem

Dataset bias causes VLN agents to overfit to seen environments and spurious correlations, hurting generalization to unseen environments.

Approach

Builds a structural causal model with observable (room references, keywords) and unobservable (style-related) confounders across vision, language, and history, and applies back-door adjustment to observable confounders and front-door adjustment (via learned confounder dictionaries from contrastively trained attentive pooling) to unobservable ones, intervening on intermediate features rather than o

Key results

• On R2R test unseen, GOAT reaches 74.57 SR and 64.94 SPL versus BEVBert's 73 SR and 62 SPL.
• On REVERIE test unseen, GOAT reports 57.72 SR and 26.70 RGSPL, ahead of BEVBert's 52.81 SR and 22.09 RGSPL.
• Ablation on R2R val-unseen: BACL alone and FACL alone each improve SR over the 75.27 baseline, and combining both reaches 77.82 SR and 68.13 SPL.

Measured effects reported in the paper

Surprising findings

• Attention-based back-door adjustment works best for text while statistic-based works best for vision, with the combined attention-attention variant performing worse than the mixed setting.

Limitations

• Intervention relies on hand-built confounder dictionaries (keywords, room types) whose quality depends on external tools like BLIP and part-of-speech tagging.
• Pre-training uses a single V100 GPU and relies on speaker-generated synthetic data and EnvEdit augmentation, so gains depend on these components.

Experimental setup

Datasets: R2R REVERIE RxR-English SOON synthetic extended datasets

Benchmarks: R2R REVERIE RxR-English SOON

Metrics: SR SPL NE OSR RGS RGSPL nDTW sDTW

Baselines: HAMT DUET TD-STP GeoVLN DSRG GridMM GELA EnvEdit BEVBert

Builds on: DUET METER BEVBert CLIP RoBERTa

Closest papers by meaning

• From Prompts to Tokens: Internalizing Causal Supervision in Vision-Language Model for Multi-Image Causal Reasoning (ICML 2026 · similarity 0.88)
• Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination Mitigation (AAAI 2026 · similarity 0.87)
• The Abstraction Gap in Vision-Language Causal Reasoning (ICML 2026 · similarity 0.86)
• Learning Invariant Causal Mechanism from Vision-Language Models (ICML 2025 · similarity 0.86)
• A Causal Marriage between VLM and IRM from Understanding to Reasoning (CVPR 2026 · similarity 0.86)
• OVER-NAV: Elevating Iterative Vision-and-Language Navigation with Open-Vocabulary Detection and StructurEd Representation (CVPR 2024 · similarity 0.86)