Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Distilling LLM Prior to Flow Model for Generalizable Agent’s Imagination in Object Goal Navigation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GOAL, a generative flow-matching model that imagines full-scene semantic maps for ObjectNav, distilling LLM-derived spatial priors into training via data-dependent couplings and Gaussian-field targets.
Problem
Object Goal Navigation requires imagining unobserved regions of indoor scenes, but prior deterministic/discriminative semantic map completion ignores layout uncertainty and generalizes poorly to unseen environments.
Approach
Use a flow-matching model trained with data-dependent couplings between partial semantic maps and LLM-enriched full-scene maps, where LLM-derived object co-occurrence distances and confidences are encoded as 2D Gaussian fields added to target maps during training only; LLM is not needed at inference.
Key results
- GOAL achieves 83.5 SR on Gibson and 41.7 SR on MP3D, outperforming SGM (78.0 and 37.7) and T-Diff on SR.
- Transfer from MP3D to HM3D yields 48.8 SR and 4.11 DTS, above PONI* (41.8 SR).
- Replacing independent Gaussian-source coupling with data-dependent coupling raises MP3D SR from 39.0 to 41.5 while reducing inference time from 17.13 ms to 12.38 ms.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add scene segmentation (ablation) | GOAL without scene segmentation or LLM prior · MP3D | SR: 32.4 -> 38.8 | Helped |
| Add LLM prior to scene segmentation (ablation) | GOAL with scene segmentation · MP3D | SR: 38.8 -> 41.7 | Helped |
| Use data-dependent coupling instead of an independent Gaussian source (ablation) | GOAL · MP3D | SR: 39.0 -> 41.5 | Helped |
| Increase model size from DiT-B to DiT-L (ablation) | GOAL with ChatGPT prior · MP3D | SR: 41.7 -> 40.5 | Hurt |
| Replace Flow Matching with MAE (ablation) | Semantics imagination model · MP3D | SR: 32.4 -> 31.9 | Hurt |
| Replace Flow Matching with MAE (ablation) | Semantics imagination model · HM3D transfer from MP3D | SR: 35.9 -> 32.1 | Hurt |
| Replace ChatGPT with ChatGLM or Deepseek (ablation) | GOAL with DiT-B · MP3D | SR: 41.5; 40.9; 41.7 | No clear effect |
| Remove LLM prior (ablation) | GOAL · HM3D transfer from MP3D | SR: 48.8 -> 47.6 | Hurt |
Surprising findings
- The choice of LLM (ChatGLM, DeepSeek, ChatGPT) has little effect on overall navigation performance, though per-scene variance is significant.
- Larger DiT-L does not improve over DiT-B and may degrade performance, attributed to overfitting.
Limitations
- Training masks are rectangular path-based approximations that differ substantially from the fan-shaped real visibility, creating a sim-to-real gap.
- Fixed-dimensional semantic map with predefined categories restricts open-vocabulary or zero-shot generalization.
- SPL gains are modest because the local policy updates long-term goals infrequently, leaving false-positive correction limited.
Experimental setup
Datasets: MP3D Gibson HM3D
Benchmarks: Gibson MP3D HM3D
Metrics: SR SPL DTS
Baselines: SemExp PONI SSC-Nav L2M Stubborn CoW 3DAware T-Diff L3MVN SG-Nav UniGoal SGM
Builds on: flow matching DiT SGM PONI Fast Marching Method
Closest papers by meaning
- LangMap: A Human-Verified Benchmark for Hierarchical Open-Vocabulary Goal Navigation (NeurIPS 2026 · similarity 0.90)
- Imagine Before Go: Self-Supervised Generative Map for Object Goal Navigation (CVPR 2024 · similarity 0.89)
- ImagineNav: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination (ICLR 2025 · similarity 0.88)
- Hierarchical Semantic-Augmented Navigation: Optimal Transport and Graph-Driven Reasoning for Vision-Language Navigation (NeurIPS 2025 · similarity 0.87)
- NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (AAAI 2024 · similarity 0.87)
- BeliefMapNav: 3D Voxel-Based Belief Map for Zero-Shot Object Navigation (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.