Research map / Detection and segmentation / 3D detection and driving
TUMTraf V2X Cooperative Perception Dataset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the TUMTraf-V2X dataset (2k point clouds, 5k images, 30k 3D boxes with track IDs from roadside and onboard sensors) and CoopDet3D, a cooperative camera-LiDAR fusion detector that beats vehicle-only fusion by +14.36 3D mAP.
Problem
Cooperative 3D object detection and tracking for V2X perception lacks real-world multi-viewpoint, multi-modal datasets with labeled tracks, since existing V2X datasets are simulated, V2V-only, or poorly documented.
Approach
A cooperative camera-LiDAR deep fusion detector that builds BEVFusion-style features separately for vehicle and infrastructure sensors and fuses them with element-wise max-pooling (PillarGrid-inspired), released alongside a real-world V2X dataset with OpenLABEL annotations and a labeling tool.
Key results
- Cooperative camera-LiDAR CoopDet3D reaches 90.76 avg 3D mAP on the south2 test set versus 76.40 for vehicle camera-LiDAR fusion, a gain of about +14.36.
- On the TUMTraf Intersection test set, CoopDet3D (infrastructure camera-LiDAR, south 2) improves average 3D mAP to 66.89 over InfraDet3D's 37.06 in the same setting.
- The selected PointPillars 512 2x + YOLOv8 s configuration runs at 11.2 FPS on an RTX 3090.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add cooperative vehicle-infrastructure camera-LiDAR fusion instead of vehicle-only camera-LiDAR perception (ablation) | CoopDet3D · TUMTraf-V2X test set, south2 FOV | mAP3D: 76.40 -> 90.76; +14.36 3D mAP | Helped |
| add cooperative perception rather than vehicle-only perception (ablation) | CoopDet3D camera-only · TUMTraf-V2X test set, south2 FOV | mAP3D: 30.36 -> 45.74 | Helped |
| add cooperative perception rather than vehicle-only perception (ablation) | CoopDet3D LiDAR-only · TUMTraf-V2X test set, south2 FOV | mAP3D: 80.11 -> 85.86 | Helped |
| add cooperative perception rather than infrastructure-only perception (ablation) | CoopDet3D camera + LiDAR · TUMTraf-V2X test set, south2 FOV | mAP3D: 87.01 -> 90.76 | Helped |
| use deep fusion instead of late fusion (ablation) | CoopDet3D vs InfraDet3D · TUMTraf Intersection test set, south 1 FOV, LiDAR | mAP3D (Avg.): 55.21 -> 69.47 | Helped |
| use deep fusion instead of late fusion (ablation) | CoopDet3D vs InfraDet3D · TUMTraf Intersection test set, south 2 FOV, LiDAR | mAP3D (Avg.): 43.13 -> 69.94 | Helped |
| use deep fusion instead of late fusion (ablation) | CoopDet3D vs InfraDet3D · TUMTraf Intersection test set, south 1 FOV, camera + LiDAR | mAP3D (Avg.): 44.55 -> 66.75 | Helped |
| use deep fusion instead of late fusion (ablation) | CoopDet3D vs InfraDet3D · TUMTraf Intersection test set, south 2 FOV, camera + LiDAR | mAP3D (Avg.): 37.06 -> 66.89 | Helped |
| use PointPillars 512 2x with YOLOv8 s instead of PointPillars 512 with YOLOv8 s (ablation) | CoopDet3D · TUMTraf-V2X cooperative detection test | mAPBEV: 94.27 -> 94.16 | No clear effect |
| retrain YOLOv8 s weights (ablation) | CoopDet3D with PointPillars 512 · TUMTraf-V2X cooperative detection test | mAPBEV: 94.27 -> 94.25 | No clear effect |
| retrain YOLOv8 s weights (ablation) | CoopDet3D with VoxelNet TS · TUMTraf-V2X cooperative detection test | mAPBEV: 92.94 -> 94.31 | Helped |
| use PointPillars 512 2x with YOLOv8 s (ablation) | CoopDet3D · TUMTraf-V2X cooperative detection test | FPS: 11.14 -> 11.20 | Helped |
Surprising findings
- Using COCO-pretrained YOLOv8 weights was not always beneficial; retraining on the dataset helped in some configurations but harmed others slightly.
- Deep feature fusion (CoopDet3D) did not beat late-fusion InfraDet3D in the LiDAR-only hard difficulty setting.
Limitations
- Offline setting; transmission bandwidth and latency not evaluated in deployment.
- Dataset covers a single intersection in Germany with limited scenes (eight 10-second scenes) and 2,000 point clouds.
- Dataset license is CC BY-NC-ND 4.0, restricting commercial and derivative use.
Experimental setup
Datasets: TUMTraf-V2X TUMTraf Intersection MS COCO
Benchmarks: TUMTraf-V2X TUMTraf Intersection
Metrics: mAP BEV mAP 3D FPS VRAM
Baselines: InfraDet3D vehicle-only camera-LiDAR fusion BEVFusion
Builds on: BEVFusion PillarGrid TransFusion 3D BAT
Closest papers by meaning
- V2X-Radar: A Multi-modal Dataset with 4D Radar for Cooperative Perception (NeurIPS 2025 · similarity 0.89)
- UrbanIng-V2X: A Large-Scale Multi-Vehicle, Multi-Infrastructure Dataset Across Multiple Intersections for Cooperative Perception (NeurIPS 2025 · similarity 0.88)
- RCooper: A Real-world Large-scale Dataset for Roadside Cooperative Perception (CVPR 2024 · similarity 0.86)
- HoloVIC: Large-scale Dataset and Benchmark for Multi-Sensor Holographic Intersection and Vehicle-Infrastructure Cooperative (CVPR 2024 · similarity 0.86)
- BEVSync: Asynchronous Data Alignment for Camera-based Vehicle-Infrastructure Cooperative Perception Under Uncertain Delays (AAAI 2025 · similarity 0.85)
- AGC-Drive: A Large-Scale Dataset for Real-World Aerial-Ground Collaboration in Driving Scenarios (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.