Research map / Label-efficient and robust learning / Test-time adaptation
Localization Is All You Evaluate: Data Leakage in Online Mapping Datasets and How to Fix It
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that nuScenes and Argoverse 2 online-mapping splits leak geographically, and proposes geographically disjoint Near and Far Extrapolation splits that reveal much lower true performance and change several design conclusions.
Problem
Standard train/val/test splits of nuScenes and Argoverse 2 revisit the same locations, so online mapping methods are evaluated on memorized locations rather than generalization to unseen areas, inflating reported performance.
Approach
Create geographically disjoint data splits (Near Extrapolation within the same cities, and Far Extrapolation via city-wise folds) for nuScenes and Argoverse 2, then re-evaluate state-of-the-art online mapping methods and re-validate prior design choices on them.
Key results
- MapTRv2 with camera input on nuScenes drops from 72.7 test mAP on the original split to 26.2 on the Near Extrapolation split (46.5 mAP drop).
- On Argoverse 2, MapTRv2 (2D) drops from 65.3 to 55.2 test mAP on the Near split, a smaller but still significant decrease.
- Re-validation shows the depth supervision benefit in MapTRv2 is not significant on nuScenes Near split, and the GKT vs BEVFormer lifter ranking flips.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train and evaluate on the geographically disjoint Near split instead of the Original split | MapTRv2 · nuScenes | mAP: 72.7 -> 26.2 | Hurt |
| train and evaluate on the geographically disjoint Near split instead of the Original split | MapTRv2 · Argoverse 2 | mAP: 65.3 -> 55.2 | Hurt |
| train and evaluate on the geographically disjoint Near split instead of the Original split | MapTRv2 · nuScenes segmentation | IoU: 31.3 -> 17.1 | Hurt |
| use dense rather than sparse training samples (ablation) | MapTRv2 · nuScenes Original validation | mAP: 61.5 -> 86.4 | Helped |
| use dense rather than sparse training samples (ablation) | MapTRv2 · nuScenes Near validation | mAP: 26.7 -> 27.2 | No clear effect |
| use dense rather than sparse training samples (ablation) | MapTRv2 · Argoverse 2 Near validation | mAP: 49.5 -> 53.1 | Helped |
| replace GKT lifting with BEVFormer (ablation) | MapTR · nuScenes Near validation | mAP: 19.0 -> 20.6 | Helped |
| replace GKT lifting with BEVFormer (ablation) | MapTR · Argoverse 2 Near validation | mAP: 48.0 -> 47.7 | No clear effect |
| add depth supervision (ablation) | MapTRv2 · nuScenes Near validation | mAP: 25.3 -> 25.9 | No clear effect |
| add depth, PV segmentation, and BEV segmentation supervision (ablation) | MapTRv2 · nuScenes Near validation | mAP: 25.3 -> 26.7 | Helped |
| add depth supervision (ablation) | MapTRv2 · Argoverse 2 Near validation | mAP: 48.8 -> 48.8 | No clear effect |
| add depth, PV segmentation, and BEV segmentation supervision (ablation) | MapTRv2 · Argoverse 2 Near validation | mAP: 48.8 -> 53.1 | Helped |
Surprising findings
- Over 80% of nuScenes validation and test samples lie within 5 m of a training sample, and about 40-45% for Argoverse 2.
- Argoverse 2 trained models generalize better than nuScenes despite greater intra-set overlap, suggesting overlap may act as beneficial augmentation.
Limitations
- Intra-set sample overlap/correlation was not addressed by the proposed splits.
- Near Extrapolation split still shares cities with training data, so it is an easier setting than far extrapolation.
- Region definitions were manual and Far Extrapolation folds are coarse with few cities.
Experimental setup
Datasets: nuScenes Argoverse 2
Benchmarks: nuScenes Argoverse 2
Metrics: mAP IoU
Baselines: VectorMapNet MapTR MapTRv2 StreamMapNet HDMapNet GKT CVT IPM
Builds on: MapTR MapTRv2 VectorMapNet StreamMapNet HDMapNet LSS
Closest papers by meaning
- TorchSpatial: A Location Encoding Framework and Benchmark for Spatial Representation Learning (NeurIPS 2024 · similarity 0.82)
- LightLoc: Learning Outdoor LiDAR Localization at Light Speed (CVPR 2025 · similarity 0.82)
- Vision-Language Reasoning for Geolocalization: A Reinforcement Learning Approach (AAAI 2026 · similarity 0.82)
- REOBench: Benchmarking Robustness of Earth Observation Foundation Models (NeurIPS 2025 · similarity 0.82)
- Beyond Chamfer Distance: Granular Order-aware Evaluation Metric For Online Mapping (NeurIPS 2026 · similarity 0.82)
- LiSA: LiDAR Localization with Semantic Awareness (CVPR 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.