Research map / 3D vision / 3D generation
LoCo: Learning 3D Location-Consistent Image Features with a Memory-Efficient Ranking Loss
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoCo, a memory-efficient reformulation and approximation of the smooth average precision ranking loss that trains a residual CNN on frozen DINO features to produce 3D location-consistent image features.
Problem
Image features that are consistent across viewpoints of the same 3D location are useful, but the smooth-AP ranking loss used to learn them needs prohibitive memory and computation for large-scale training.
Approach
Reformulates smooth AP over patch pairs with batch correction factors, subsamples anchor pairs, and drops pair similarity differences that saturate the sigmoid (two-sided, with correction terms), cutting memory by about three orders of magnitude.
Key results
- LoCo reaches 61.8% recall at 0–15° viewpoint change on Paired ScanNet correspondence vs 45.0% for DINO and 23.5% for LoCUS.
- The memory-efficient loss allows batches about 2000x larger than the LoCUS setting, with the pair matrix reduced from 5.7GB to about 500KB.
- On scene-stable panoptic segmentation, LoCo gets 66.5 Jac / 67.5 IoU on Matterport3D, outperforming DINO (65.2 / 65.5) and comparable to CroCo-v2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train location-consistent features with LoCo | DINO ViT-Base features · ScanNet Paired pixel correspondence, 0°–15° | recall: 45.0 -> 61.8 | Helped |
| train location-consistent features with LoCo | DINO ViT-Base features · ScanNet Paired pixel correspondence, 60°–180° | recall: 11.2 -> 10.3 | Hurt |
| train LoCo features | DINO ViT-Base features · Matterport3D scene-stable segmentation | IoU: 65.5 -> 67.5 | Helped |
| train LoCo features | DINO ViT-Base features · ScanNet scene-stable segmentation | AP: 89.4 -> 88.8 | Hurt |
| train LoCo features | DINO ViT-Base features · Matterport3D VPR | Recall@1: 45.7 -> 46.8 | Helped |
| train LoCo features | DINO ViT-Base features · ScanNet VPR | Recall@5: 92.9 -> 92.5 | Hurt |
| decrease saturation threshold ∆ from 0.076 to 0.053 (ablation) | LoCo on ScanNet Paired · 0°–15° viewpoint change | recall: 61.8 -> 59.9 | Hurt |
| decrease saturation threshold ∆ from 0.076 to 0.029 (ablation) | LoCo on ScanNet Paired · 0°–15° viewpoint change | recall: 61.8 -> 57.9 | Hurt |
| use a DINOv2 backbone instead of a DINO backbone (ablation) | LoCo · ScanNet Paired pixel correspondence, 0°–15° | recall: 61.8 -> 58.2 | Hurt |
| use a DINOv2 backbone instead of a DINO backbone (ablation) | LoCo · Matterport3D scene-stable segmentation | Jac: 66.5 -> 61.1 | Hurt |
| use the LoCUS architecture (ablation) | LoCo training algorithm · ScanNet Paired pixel correspondence, 0°–15° | recall: 61.8 -> 27.8 | Hurt |
| use the LoCUS architecture (ablation) | LoCo training algorithm · ScanNet Paired pixel correspondence, 60°–180° | recall: 10.3 -> 6.9 | Hurt |
Surprising findings
- Training the original LoCUS architecture with the LoCo loss performs worse than original LoCUS on panoptic segmentation, so memory savings alone do not improve performance.
- DINOv2 final-layer features are less location-consistent than DINO, while its first six blocks are better on correspondence.
Limitations
- Requires camera poses and depth maps for training images.
- Only trains small CNNs or fine-tunes few layers on frozen foundation features; not tested for training foundation models from scratch.
- Experiments use a small training set of about 59 environments; scaling laws not investigated.
Experimental setup
Datasets: Matterport3D ScanNet Paired ScanNet split
Benchmarks: Paired ScanNet pixel correspondence Matterport3D scene-stable panoptic segmentation ScanNet scene-stable panoptic segmentation Matterport3D VPR ScanNet VPR
Metrics: recall at 10px reprojection error Jaccard index IoU AP Recall@1 Recall@5 GPU memory
Baselines: LoCUS DINO DINOv2 CroCo-v2 MaskDINO MixVPR NCE with hard mining
Builds on: smooth average precision (Smooth-AP) LoCUS DINO DINO-Tracker
Closest papers by meaning
- Near, far: Patch-ordering enhances vision foundation models' scene understanding (ICLR 2025 · similarity 0.87)
- Tango3D: Towards Alignment for Global and Local 2D-3D Correspondence (NeurIPS 2026 · similarity 0.86)
- OneCanvas: 3D Scene Understanding via Panoramic Reprojection (NeurIPS 2026 · similarity 0.85)
- Multi-view Consistent 3D Panoptic Scene Understanding (AAAI 2025 · similarity 0.85)
- Bootstraping Clustering of Gaussians for View-consistent 3D Scene Understanding (AAAI 2025 · similarity 0.84)
- GLACE: Global Local Accelerated Coordinate Encoding (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.