Research map / Vision-language and multimodal / Robot action and navigation
DiaLoc: An Iterative Approach to Embodied Dialog Localization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiaLoc, an iterative transformer-based localizer that predicts the observer's location heatmap after each dialog turn on the WAY embodied dialog localization dataset.
Problem
Embodied dialog-based localization has been studied with methods that need the complete dialog before predicting, which is impractical for deployed systems where location estimates should update during the conversation.
Approach
Replaces the full-dialog image-to-image LingUNet approach with a ViT map encoder and a BERT-conditioned multimodal transformer fusion encoder that updates a hidden state turn by turn, producing per-turn location heatmaps with an auxiliary diversity loss and decayed multi-shot loss.
Key results
- Single-shot DiaLoc improves Acc5 on valUnseen by 7.08 points over LingUNet (40.41 vs 33.33 for BLIP(DiaLoc) vs LingUNet baseline reported in table).
- DiaLoc-e reaches 47.15 Acc5 on valUnseen in multi-shot mode versus 36.30 for LingUNet-e, a gain of about 10.85 points.
- GPT paraphrase augmentation lowers valUnseen LE from 9.21 to 9.09 and raises valUnseen Acc5 from 37.95 to 40.71 for DiaLoc-e.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DiaLoc instead of LingUNet for single-shot localization | WAY localization · valUnseen | Acc5: +7.08 | Helped |
| use DiaLoc-e instead of LingUNet-e for multi-shot localization | WAY localization · valUnseen | Acc5: 36.30 -> 47.15 | Helped |
| use fusion encoder depth 3 instead of depth 1 (ablation) | DiaLoc-i with frozen ViT · WAY valUnseen | Acc5: 33.27 -> 38.23 | Helped |
| use fusion encoder depth 3 instead of depth 1 (ablation) | DiaLoc-i with fine-tuned ViT · WAY valUnseen | Acc5: 47.09 -> 42.06 | Hurt |
| use implicit instead of explicit fusion (ablation) | DiaLoc, depth 3 · WAY valSeen | Acc5: 50.32 -> 58.01 | Helped |
| use implicit instead of explicit fusion (ablation) | DiaLoc, depth 3 · WAY valUnseen | Acc5: 41.60 -> 42.06 | Hurt |
| set decay factor to 0.5 instead of 0.0 (ablation) | DiaLoc-i, depth 3 · WAY valUnseen | Acc5: 42.06 -> 36.48 | Hurt |
| add the auxiliary loss (ablation) | DiaLoc-e, decay 0.0 · WAY valUnseen | Acc5: 37.89 -> 40.23 | Helped |
| add the auxiliary loss (ablation) | DiaLoc-i, decay 0.5 · WAY valUnseen | Acc5: 36.41 -> 32.53 | Hurt |
| add GPT-paraphrased training dialogs (ablation) | DiaLoc-e · WAY valUnseen | Acc5: 37.95 -> 40.71 | Helped |
| add GPT-paraphrased training dialogs (ablation) | DiaLoc-e · WAY valSeen | Acc5: 51.56 -> 60.00 | Helped |
| use more dialog turns for iterative localization | DiaLoc · WAY valSeen and valUnseen | localization error (LE) | Helped |
Surprising findings
- Multi-shot LingUNet performs well on valUnseen even though it drops on valSeen, suggesting the iterative approach reduces overfitting.
- Adding more dialog turns does not consistently improve localization for dialogs with T=6 turns, possibly due to unbalanced training data.
Limitations
- Evaluation uses known floor and dense per-turn locations, which are not available in the test split, so the setting is simplified.
- Multi-shot predictions use argmax only, which is unfair when multiple peaks exist.
- Performance degrades on longer dialogs (T=6), and more turns do not always help.
Experimental setup
Datasets: WAY
Benchmarks: WAY
Metrics: Acc0 Acc5 localization error (geodesic) CMC curves
Baselines: LingUNet BLIP-Res18 BLIP(DiaLoc) LingUNet-i LingUNet-e
Builds on: LingUNet BLIP Vision Transformer BERT
Closest papers by meaning
- ERGeoBench: A Comprehensive Benchmark for Embodied Reasoning and Geo-localization in Multimodal Large Language Models (ICML 2026 · similarity 0.88)
- Towards Precise Embodied Dialogue Localization via Causality Guided Diffusion (CVPR 2025 · similarity 0.88)
- NavBench: Probing Multimodal Large Language Models for Embodied Navigation (NeurIPS 2025 · similarity 0.87)
- DialogueVPR: Towards Conversational Visual Place Recognition (CVPR 2026 · similarity 0.86)
- MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI (ICML 2024 · similarity 0.85)
- Mechanisms of Object Localization in Vision-Language Models (CVPR 2026 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.