Research map / 3D vision / 3D generation
RoboSense: Large-scale Dataset and Benchmark for Egocentric Robot Perception and Navigation in Crowded and Unstructured Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RoboSense, a large-scale multimodal egocentric dataset (camera, fisheye, LiDAR) from a sweeper robot in crowded, unstructured outdoor scenes, with six benchmark tasks and baselines for detection, tracking, motion prediction and occupancy.
Problem
Existing perception and prediction benchmarks target autonomous driving on roads and highways, leaving egocentric near-field perception for mobile robots navigating crowded, unstructured environments under-served.
Approach
A 360-degree multi-sensor robot platform (4 cameras, 4 fisheye, 4 LiDAR) collects near-field-dense annotated data with track IDs, plus a closest-collision-point distance proportion matching criterion for near-field 3D detection and prediction metrics.
Key results
- Transfusion-L reaches 83.6 Vehicle 3D AP at p=5% under center-point matching, dropping to 65.1 under closest-collision-point matching on LiDAR detection.
- Multi-modal late fusion (8V+4L) raises CCP-based 3D AP within 5m from 20.5% (4C+4F vision) to 36.9%.
- PnPNet achieves minADE 0.89 m and minFDE 1.12 m, better than ViP3D (1.31 / 1.55) on RoboSense motion forecasting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace PointPillar with Transfusion-L for LiDAR 3D detection | RoboSense LiDAR 3D detection · RoboSense validation, Vehicle 3D AP | 3D AP: 72.5/53.0 -> 83.6/65.1 | Helped |
| Add temporal clues with BEVDet4D (ablation) | BEVDet · RoboSense validation, Multi-view 3D Detection | 3D AP: 76.2/30.2 -> 77.2/31.1; 42.3/25.7 -> 42.0/24.8; 47.4/28.5 -> 48.1/29.3 | Mixed |
| Add an additional depth prediction branch with point supervision (ablation) | BEVDet · RoboSense validation, Multi-view 3D Detection | 3D AP: 76.2/30.2 -> 77.8/31.3; 42.3/25.7 -> 43.3/27.0; 47.4/28.5 -> 50.1/31.3 | Helped |
| Combine 4C and 4F camera layouts instead of using either alone (ablation) | BEVDepth multi-view detection · RoboSense validation, 3D AP across [0, 5], [5, 10], and [10, 30]m | 3D AP: 54.9/16.0 -> 68.9/20.5; 60.1/18.3 -> 75.2/22.9; 53.7/33.1 -> 64.2/38.6 | Helped |
| Use multi-modal 8V+4L input instead of 4C+4F cameras (ablation) | AB3DMOT · RoboSense validation, 3D MOT | sAMOTA: 51.16 -> 43.32 | Hurt |
| Use PnPNet instead of ViP3D for motion forecasting | RoboSense motion forecasting · RoboSense validation | minADE (m): 1.31 -> 0.89 | Helped |
| Replace constant-position predictions with PnPNet | Motion forecasting with ground-truth boxes · RoboSense validation | minADE (m): 2.42 -> 0.89 | Helped |
Surprising findings
- Closest-collision-point matching lowers 3D AP substantially versus center-point matching (e.g., 18.5% drop for Transfusion-L on Vehicle), indicating near-field localization is much harder.
- Vision-based 3D detection on near-field objects within 5m is weaker than LiDAR detection (19.3% vs 20.5% 3D AP under CCP), contrary to the usual LiDAR advantage.
Limitations
- Data collected at a single location (Dishui Lake, Shanghai) with 22 places and a single robot platform.
- Annotation limited to three movable classes (Vehicle, Cyclist, Pedestrian) at 1Hz keyframes.
- Occupancy instance IDs and motion vectors left to future work.
Experimental setup
Datasets: RoboSense
Benchmarks: RoboSense
Metrics: 3D AP AOS ASE sAMOTA AMOTP minADE minFDE MR EPA mIoU
Baselines: PointPillar SECOND PV-RCNN Transfusion-L BEVDet BEVDet4D BEVDepth BEVFormer ViP3D PnPNet
Builds on: KITTI nuScenes Waymo Open Dataset AB3DMOT BEVDepth
Closest papers by meaning
- The Road Ahead in Autonomous Driving: The KITScenes Multimodal Dataset (NeurIPS 2026 · similarity 0.86)
- RoboSpatial: Teaching Spatial Understanding to 2D and 3D Vision-Language Models for Robotics (CVPR 2025 · similarity 0.85)
- Seeing Together: Multi-Robot Cooperative Egocentric Spatial Reasoning with Multimodal Large Language Models (NeurIPS 2026 · similarity 0.85)
- MCD: Diverse Large-Scale Multi-Campus Dataset for Robot Perception (CVPR 2024 · similarity 0.85)
- EgoGen: An Egocentric Synthetic Data Generator (CVPR 2024 · similarity 0.85)
- Transfer Your Perspective: Controllable 3D Generation from Any Viewpoint in a Driving Scene (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.