Research map / Point clouds / Point cloud classification
Text2Loc: 3D Point Cloud Localization from Natural Language
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Text2Loc localizes 3D point cloud submaps and positions from natural-language descriptions using a coarse-to-fine pipeline (text-submap retrieval with HTM and contrastive learning, then matching-free regression).
Problem
Localizing a position in a city-scale 3D point cloud map from a few natural-language spatial descriptions (the "last mile problem"), where prior methods relied on costly text-instance matching and pairwise ranking losses.
Approach
A hierarchical transformer with max-pooling (HTM) encodes intra- and inter-sentence relations on top of a frozen T5, is trained with text-submap contrastive loss, and fine localization drops text-instance matching in favor of prototype-based map cloning and cascaded cross-attention to regress the target position directly.
Key results
- Text2Loc reaches 0.32 top-1 submap retrieval recall on validation and 0.28 on test, versus 0.14/0.12 for Text2Pos.
- Fine localization top-1 recall at <5m is 0.33 on test, about 2x the previous state of the art RET (0.16 at <5m) and Text2Pos (0.13).
- The matching-free fine localization uses 1.06M parameters and 2.27 ms runtime versus 2.08M and 43.11 ms for the matcher variant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Text2Loc instead of RET | KITTI360Pose localization · KITTI360Pose test | Localization recall (ϵ < 5m), k=1: 0.16 -> 0.33 | Helped |
| use Text2Loc instead of RET | KITTI360Pose global place recognition · KITTI360Pose validation | Submap retrieval recall, k=1: 0.18 -> 0.32 | Helped |
| add frozen pre-trained T5 instead of the Text2Pos LSTM (ablation) | Text2Loc global place recognition · KITTI360Pose test | Submap retrieval recall, k=1: 0.26 -> 0.28 | Helped |
| add hierarchical transformer with max-pooling (ablation) | Text2Loc global place recognition · KITTI360Pose test | Submap retrieval recall, k=1: 0.28 -> 0.28 | No clear effect |
| add hierarchical transformer with max-pooling (ablation) | Text2Loc global place recognition · KITTI360Pose validation | Submap retrieval recall, k=1: 0.30 -> 0.32 | Helped |
| use contrastive learning instead of pairwise ranking loss (ablation) | Text2Loc global place recognition · KITTI360Pose test | Submap retrieval recall, k=1: 0.20 -> 0.28 | Helped |
| add number encoder (ablation) | Text2Loc global place recognition · KITTI360Pose validation | Submap retrieval recall, k=1: 0.30 -> 0.32 | Helped |
| add CCAT while using the same retrieved submaps (ablation) | Text2Loc fine localization · KITTI360Pose test | Localization recall (ϵ < 5m), k=1: 0.30 -> 0.32 | Helped |
| add PMC to CCAT-based matching-free fine localization (ablation) | Text2Loc fine localization · KITTI360Pose test | Localization recall (ϵ < 5m), k=1: 0.32 -> 0.33 | Helped |
| replace text-instance matcher with matching-free CCAT | Text2Loc fine localization · KITTI360Pose test | Runtime (ms): 43.11 -> 2.27 | Helped |
Surprising findings
- Modifying a single sentence in the query drops top-1 submap retrieval recall from 0.28 to 0.15, showing high sensitivity to text changes.
Limitations
- Performance depends heavily on the coarse retrieval stage; failure occurs when all retrieved submaps are negative.
- Sensitive to text changes, with a single sentence edit causing large recall drops.
- Evaluated only on KITTI360Pose; low diversity of outdoor scenes causes confusable submaps.
Experimental setup
Datasets: KITTI360Pose
Benchmarks: KITTI360Pose
Metrics: submap retrieval recall at top-k localization recall at error thresholds 5/10/15m parameters (M) runtime (ms)
Baselines: Text2Pos RET Text2Pos fine localization with matcher pairwise ranking loss variant
Builds on: Text2Pos RET CLIP PointNet++ CASSPR
Closest papers by meaning
- Text to Point Cloud Localization with Multi-Level Negative Contrastive Learning (AAAI 2025 · similarity 0.92)
- CMMLoc: Advancing Text-to-PointCloud Localization with Cauchy-Mixture-Model Based Framework (CVPR 2025 · similarity 0.90)
- More Text, Less Point: Towards 3D Data-Efficient Point-Language Understanding (AAAI 2025 · similarity 0.85)
- LOCATE 3D: Real-World Object Localization via Self-Supervised Learning in 3D (ICML 2025 · similarity 0.85)
- LocLLM: Exploiting Generalizable Human Keypoint Localization via Large Language Model (CVPR 2024 · similarity 0.85)
- Recognition through Reasoning: Reinforcing Image Geo-localization with Large Vision-Language Models (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.