Research map / 3D vision / 3D generation
LOCATE 3D: Real-World Object Localization via Self-Supervised Learning in 3D
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LOCATE 3D, a sensor-based 3D referring-expression grounding model that pretrains a PTv3 encoder on lifted 2D foundation features with a latent masked-prediction SSL objective (3D-JEPA), then fine-tunes a language-conditioned mask-and-box decoder, and releases the L3DD dataset.
Problem
3D referring expression localization from raw posed RGB-D sensor streams, without ground-truth meshes, instance segmentation, or region proposals, which is needed for real-world robot and AR deployment.
Approach
Lift DINOv2 and CLIP (per-SAM-mask) features into a voxelized 3D point cloud, pretrain a PointTransformerV3 encoder with JEPA-style masked prediction in latent space (3D-JEPA) to obtain scene-contextualized features, and decode referred objects with a language-conditioned decoder that jointly predicts masks and boxes.
Key results
- Locate 3D reaches 61.7% Acc@25 on joint SR3D/NR3D/ScanRefer evaluation from sensor point clouds, versus 58.5% for PQ3D (mesh input) and 40.7% for BUTD-DETR (sensor input).
- 3D-JEPA pretraining improves in-domain joint Acc@25 from 59.8% to 61.7% and out-of-domain ARKitScenes/ScanNet++ generalization from 51.5% to 56.7% on SN++.
- Locate 3D+ trained with L3DD data reaches 63.7% Acc@25 jointly; on the Spot robot task it succeeds 8/10 trials versus 5.66/10 for the best VLM baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pre-train the encoder with 3D-JEPA and fine-tune it for localization | Locate 3D with CF features · Joint evaluation (SR3D, NR3D, ScanRefer) | Acc@25: 59.8% → 61.7% | Helped |
| Add L3DD training data | Locate 3D · Joint evaluation (SR3D, NR3D, ScanRefer) | Acc@25: 61.7% → 63.7% | Helped |
| Replace RGB inputs with lifted CF features (ablation) | 3D referential grounding model · Joint evaluation (SR3D, NR3D, ScanRefer) | Acc@25: 28.9% → 53.9% | Helped |
| Add an encoder with random initialization to CF features (ablation) | 3D referential grounding model · Joint evaluation (SR3D, NR3D, ScanRefer) | Acc@25: 53.9% → 59.8% | Helped |
| Fine-tune rather than freeze the 3D-JEPA encoder (ablation) | Locate 3D · Joint evaluation (SR3D, NR3D, ScanRefer) | Acc@25: 56.2% → 61.7% | Helped |
| Combine CLIP-L (SAM-H) with DINO-v2 features (ablation) | Locate 3D · Joint evaluation (SR3D, NR3D, ScanRefer) | Acc@25: 59.2 → 61.7 | Helped |
| Add box supervision to mask supervision (ablation) | Locate 3D decoder · Joint evaluation (SR3D, NR3D, ScanRefer) | Acc@25: 55.4% → 61.7% | Helped |
| Replace the MLP box head with the transformer box head (ablation) | Locate 3D decoder · Joint evaluation (SR3D, NR3D, ScanRefer) | Acc@25: 35.6% vs 61.7% | Helped |
| Remove mask supervision (ablation) | Locate 3D decoder · Joint evaluation (SR3D, NR3D, ScanRefer) | Acc@25: 61.7 vs 0.3 | Hurt |
| Scale the decoder from Small to Large (ablation) | Locate 3D with 3D-JEPA initialization · Joint evaluation (SR3D, NR3D, ScanRefer) | Acc@25: 58.1 → 61.7 | Helped |
| Use a stage-wise rather than naive learning-rate schedule (ablation) | 3D-JEPA initialized Locate 3D · Joint evaluation (SR3D, NR3D, ScanRefer) | Acc@25: 59.5 → 61.7 | Helped |
| Train on additional ScanNet++ scenes rather than the same quantity of ScanNet samples (ablation) | Locate 3D trained with 30K L3DD samples · Joint evaluation | Joint Evaluation: 0.618 → 0.632 | Helped |
Surprising findings
- Mask-only supervision with DBSCAN post-processing is a competitive alternative, and box-only supervision collapses to 0.3% Acc@25.
- Adding in-domain L3DD training data reduces the benefit of 3D-JEPA pretraining on L3DD evaluations (ScanNet+LX3D JEPA vs random init are close).
Limitations
- Assumes static or quasi-static environments; caching of 2D features does not handle dynamic scenes.
- Evaluated on sensor point clouds that are noisier than mesh point clouds, with performance degradation due to sensor noise and registration errors.
- Relies on a stage-wise training schedule and substantial pretraining design choices, with out-of-domain gains partly reduced by in-domain data.
Experimental setup
Datasets: ScanNet ScanNet++ ARKitScenes SR3D NR3D ScanRefer
Benchmarks: SR3D NR3D ScanRefer ScanNet++ (L3DD) ARKitScenes (L3DD) FRE held-out apartment
Metrics: top-1 accuracy@25 IoU top-1 accuracy@50 IoU task success rate (robot)
Baselines: BUTD-DETR 3D-VisTA PQ3D ConcreteNet 3D-LLM ConceptFusion GPT-4o VLM baseline Llama-3 VLM baseline
Builds on: JEPA Point Transformer v3 ConceptFusion BUTD-DETR DINOv2
Closest papers by meaning
- Leveraging Imagery Data with Spatial Point Prior for Weakly Semi-supervised 3D Object Detection (AAAI 2024 · similarity 0.88)
- From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenes (NeurIPS 2025 · similarity 0.87)
- Sonata: Self-Supervised Learning of Reliable Point Representations (CVPR 2025 · similarity 0.87)
- Rooms from Motion: Un-posed Indoor 3D Object Detection as Localization and Mapping (NeurIPS 2025 · similarity 0.87)
- MonoPlace3D: Learning 3D-Aware Object Placement for 3D Monocular Detection (CVPR 2025 · similarity 0.86)
- M3SOT: Multi-Frame, Multi-Field, Multi-Space 3D Single Object Tracking (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.