Research map / Vision-language and multimodal / Robot action and navigation
Augmented Commonsense Knowledge for Remote Object Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Augments the DUET vision-and-language navigation agent on REVERIE with ConceptNet commonsense knowledge organized as a spatio-temporal knowledge graph, plus a concept history and commonsense-based decision pipeline.
Problem
Remote object grounding in VLN (REVERIE) with concise high-level instructions, where viewpoint representations from whole-image or object features are insufficient for accurate action prediction.
Approach
Retrieve ConceptNet triples for detected objects, refine them with CLIP similarity, and model objects, knowledge, and concept history as a spatio-temporal knowledge graph via a knowledge graph-aware cross-modal encoder and concept aggregation, plus a new local action pipeline added to DUET during fine-tuning.
Key results
- On REVERIE test unseen, ACK reaches SR 53.97 and SPL 37.89, versus DUET's 52.51 and 36.06.
- On REVERIE test unseen, ACK achieves RGS 32.77 and RGSPL 23.15, improving over DUET's 31.88 and 22.06.
- Ablation on validation unseen shows top-10 retrieved knowledge works best; top-20 degrades performance.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add commonsense knowledge and ACK modules to DUET | DUET · REVERIE test unseen | SR: 52.51 -> 53.97 | Helped |
| add commonsense knowledge and ACK modules to DUET | DUET · REVERIE test unseen | RGS: 31.88 -> 32.77 | Helped |
| add commonsense knowledge and ACK modules to DUET | DUET · REVERIE validation unseen | SR: 46.98 -> 47.49 | Helped |
| add knowledge graph-aware self-attention (ablation) | ACK · REVERIE validation unseen | SR: 45.92 -> 46.18 | Helped |
| add concept history (ablation) | ACK with KGS · REVERIE validation unseen | SR: 46.18 -> 48.08 | Helped |
| add commonsense-based decision-making pipeline (ablation) | ACK with KGS and concept history · REVERIE validation unseen | SPL: 34.02 -> 34.44 | Helped |
| increase external knowledge from top-k 0 to top-k 10 (ablation) | ACK · REVERIE validation unseen | OSR: 51.49 -> 52.77 | Helped |
| increase external knowledge from top-k 10 to top-k 20 (ablation) | ACK · REVERIE validation unseen | OSR: 52.77 -> 51.66 | Hurt |
| incorporate commonsense knowledge into ACK | ACK · R2R | performance | No clear effect |
Surprising findings
- Commonsense knowledge gave no significant improvement on R2R, which the authors attribute to fine-grained instructions making exploration less essential.
- Adding more retrieved knowledge (k=20) hurt performance relative to k=10, which the authors attribute to noise.
Limitations
- Gains over DUET are small (about 1-2 points) and SR on validation unseen is lower than KERM's.
- R2R results show no significant improvement from commonsense knowledge.
- Relies on a single external knowledge source (ConceptNet) and object labels rather than visual features for knowledge nodes.
Experimental setup
Datasets: REVERIE ConceptNet Visual Genome ImageNet
Benchmarks: REVERIE R2R
Metrics: OSR SR SPL RGS RGSPL TL
Baselines: DUET KERM AZHP HAMT TD-STP HOP RecBERT Airbert SIA CKR
Builds on: DUET ConceptNet CLIP Faster R-CNN
Closest papers by meaning
- Room-and-Object Aware Knowledge Reasoning for Remote Embodied Referring Expression (CVPR 2021 · similarity 0.89)
- NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (AAAI 2024 · similarity 0.86)
- Hierarchical Semantic-Augmented Navigation: Optimal Transport and Graph-Driven Reasoning for Vision-Language Navigation (NeurIPS 2025 · similarity 0.86)
- FantasyVLN: Unified Multimodal Chain-of-Thought Reasoning for Vision-and-Language Navigation (CVPR 2026 · similarity 0.86)
- DreamVLA: A Vision-Language-Action Model Dreamed with Comprehensive World Knowledge (NeurIPS 2025 · similarity 0.86)
- ESCA: Contextualizing Embodied Agents via Scene-Graph Generation (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.