Research map / 3D vision / 3D generation
g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Pre-trains a generalizable 3D-language feature field model (g3D-LF) on ~5K scenes with multi-level contrastive learning, then plugs its novel-view, panorama and BEV representations into VLN, zero-shot object navigation and situated QA.
Problem
Generalizable 3D feature fields used for embodied tasks are supervised only by 2D foundation models (CLIP, DINOv2), which limits 3D spatial understanding and language alignment, especially for long text over large-scale panorama and BEV representations.
Approach
Pre-train a feature-field model on a curated ~1M-annotation 3D-language dataset with multi-level contrastive losses: balanced object-level alignment over 1,883 categories, fine-grained BEV-window-to-long-text affinity alignment, and CLIP distillation from ground-truth views.
Key results
- Monocular VLN on R2R-CE test unseen: SR 46.3 and SPL 32.2, improving over VLN-3DFF (43.7 SR).
- Zero-shot object navigation on HM3D: SR 55.6 and SPL 31.8 versus VLFM's 52.5 SR and 30.4 SPL, with feature fields replacing BLIP-2 for querying.
- Panorama VLN on R2R-CE val unseen reaches SPL 52 (SOTA on SPL) while SR 61 is comparable to HNR at 61.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use g3D-LF representations instead of forward-view CLIP features and BLIP-2 mapping (ablation) | Monocular VLN and object navigation · R2R-CE val unseen; HM3D val | SR: 40.7 -> 47.2; 34.2 -> 55.6 | Helped |
| Add novel-view/panorama representations (ablation) | g3D-LF embodied tasks · R2R-CE val unseen, monocular | SR: 42.8 -> 47.2 | Helped |
| Add BEV map representations (ablation) | g3D-LF embodied tasks · R2R-CE val unseen, monocular | SR: 46.9 -> 47.2 | Helped |
| Add novel-view/panorama representations (ablation) | g3D-LF object navigation · HM3D val | SR: 50.2 -> 55.6 | Helped |
| Add BEV map representations (ablation) | g3D-LF object navigation · HM3D val | SR: 53.9 -> 55.6 | Helped |
| Add multi-level contrastive pre-training (ablation) | g3D-LF · HM3D val | SR: 34.2 -> 55.6 | Helped |
| Add object-level and fine-grained language supervision alongside CLIP distillation (ablation) | g3D-LF · HM3D val | SR: 47.6 -> 55.6 | Helped |
| Add CLIP visual-feature distillation (ablation) | g3D-LF · R2R-CE val unseen, monocular | SR: 42.4 -> 47.2 | Helped |
| Use balanced loss weighting for hard-to-recognize objects (ablation) | g3D-LF object navigation · HM3D val | SR: 51.7 -> 55.6 | Helped |
| Use fine-grained long-text contrastive learning instead of a coarse [SEP] vector (ablation) | g3D-LF · R2R-CE val unseen, monocular | SR: 45.7 -> 47.2 | Helped |
| Replace VLN-3DFF with g3D-LF representations | Monocular VLN · R2R-CE Test Unseen | SR: 43.7 -> 46.3 | Helped |
| Replace HNR with g3D-LF representations | Panorama VLN · R2R-CE Test Unseen | SPL: 50 -> 51 | Helped |
Surprising findings
- Language supervision alone performs poorly; CLIP distillation is needed alongside language supervision for good VLN results.
- Replacing BLIP-2 with g3D-LF in VLFM improves SR by about 3 points on HM3D while also requiring no VLM for querying.
Limitations
- Cannot adapt to dynamic environments with moving objects or people.
- Not evaluated on dynamic tasks such as object manipulation.
- 3D-language training data scale and quality remain limited; SQA3D answer accuracy trails LLM-based methods.
Experimental setup
Datasets: ScanNet HM3D Structured3D SceneVerse R2R-CE HM3D object navigation val MP3D object navigation val SQA3D
Benchmarks: R2R-CE HM3D ObjectNav MP3D ObjectNav SQA3D
Metrics: NE OSR SR SPL EM@1 Acc@0.5m Acc@1.0m Acc@15° Acc@30° FPS
Baselines: VLN-3DFF CM2 WS-MGMap NaVid InstructNav HNR ETPNav BEVBert VLFM SG-Nav LEO Scene-LLM
Builds on: HNR VLN-3DFF VLFM NeRF GridMM
Closest papers by meaning
- GA-VLN: Geometry-Aware BEV Representation for Efficient Vision-Language Navigation (CVPR 2026 · similarity 0.89)
- G^3-LQ: Marrying Hyperbolic Alignment with Explicit Semantic-Geometric Modeling for 3D Visual Grounding (CVPR 2024 · similarity 0.89)
- An Embodied Generalist Agent in 3D World (ICML 2024 · similarity 0.89)
- Unifying 2D and 3D Vision-Language Understanding (ICML 2025 · similarity 0.89)
- Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding (CVPR 2025 · similarity 0.89)
- D3D-VLP: Dynamic 3D Vision-Language-Planning Model for Embodied Grounding and Navigation (CVPR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.