Research map / Multi-agent systems and games / Human-AI and goal-conditioned
SubT-MRS Dataset: Pushing SLAM Towards All-weather Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SubT-MRS, a real-world multimodal, multi-robot, multi-degradation SLAM dataset (5 years, 500+ hours, 100+ km) plus a velocity-based AUC-F1 robustness metric, benchmarked via an ICCV'23 SLAM challenge.
Problem
Existing SLAM datasets are collected in controlled conditions and lack all-weather degradations (darkness, smoke, dust, snow, textureless structures) and a metric that captures robustness and velocity reliability, limiting robust SLAM development.
Approach
A dataset spanning diverse degraded environments, heterogeneous robots (aerial, legged, wheeled) and synchronized multimodal sensors with centimeter-level ground truth, paired with a robustness metric defined as the AUC of an F-1 score over estimated linear and angular velocity errors.
Key results
- Best LiDAR-track ATE was 0.588 m (FAST-LIO2 + HBA, Liu et al.), at 51.3 s per iteration; best visual-track ATE was 0.547 m (DVI-SLAM, Peng et al.), at 183 s.
- Average robustness Rp/Rr for mixed-degradation LiDAR sequences (0.290, 0.473) is much lower than for geometric degradation (0.664, 0.754), supporting the metric's ability to separate difficulty levels.
- From 29 challenge submissions, 5 LiDAR and 4 visual winners were identified; no sensor-fusion submission met the success criteria.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate SLAM with the proposed velocity-based robustness metric rather than ATE alone | SLAM Challenge methods · Mixed degradation | Rp: 0.664 to 0.290 | Hurt |
| Evaluate SLAM with the proposed velocity-based robustness metric rather than ATE alone | SLAM Challenge methods · Mixed degradation | Rr: 0.754 to 0.473 | Hurt |
| Run visual SLAM in low-quality images, including smoke and nighttime conditions | Visual SLAM methods · Smoke Room and Outdoor Night | ATE (m): 4.95m, 7.76m | Hurt |
| Run LiDAR SLAM in simulated sequences with aggressive and random motion | LiDAR SLAM methods · Simulation | ATE (m): 5.239m compared to real-world's (0.836m) | Hurt |
| Use sensor-fusion SLAM in mixed-degradation scenarios | SLAM Challenge methods · Mixed degradation | ATE (m): 1.61m compared with 0.836m | Hurt |
Surprising findings
- No submission in the sensor-fusion track met the success criteria, indicating existing SLAM systems remain far from robust in mixed degradation.
- The accuracy ranking (ATE) differs from the robustness ranking, since the F-1-based metric jointly considers recall and precision.
Limitations
- Simulated sequences derived from TartanAir have a sim-to-real gap and were not thoroughly tested by SLAM methods.
- Existing LiDAR and visual methods struggle with caves, smoke, night, and aggressive motion, and no current solution balances accuracy and real-time performance.
- Robustness metric requires B-spline smoothing when algorithms do not output velocity at the desired frequency.
Experimental setup
Datasets: SubT-MRS
Benchmarks: SubT-MRS
Metrics: ATE RPE robustness metric (AUC of F-1 score, Rp and Rr)
Baselines: FAST-LIO2 DVI-SLAM VINS-Mono ORB-SLAM3 DLO LIO-EKF Point-LIO VR-SLAM
Builds on: SubT challenge Kalibr Super Odometry GICP
Closest papers by meaning
- DeepPointMap: Advancing LiDAR SLAM with Unified Neural Descriptors (AAAI 2024 · similarity 0.80)
- USCILab3D: A Large-scale, Long-term, Semantically Annotated Outdoor Dataset (NeurIPS 2024 · similarity 0.80)
- Weatherproofing Retrieval for Localization with Generative AI and Geometric Consistency (ICLR 2024 · similarity 0.79)
- MCD: Diverse Large-Scale Multi-Campus Dataset for Robot Perception (CVPR 2024 · similarity 0.79)
- Weather-Robust Cross-View Geo-Localization via Prototype-Based Semantic Part Discovery (NeurIPS 2026 · similarity 0.79)
- SDTagNet: Leveraging Text-Annotated Navigation Maps for Online HD Map Construction (NeurIPS 2025 · similarity 0.79)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.