Research map / 3D vision / 3D generation
Multi-View Pedestrian Occupancy Prediction with a Novel Synthetic Dataset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MVP-Occ, a synthetic CARLA-based multi-view pedestrian occupancy dataset with voxel semantic/instance labels, and OmniOcc, a baseline that predicts 2D pedestrian occupancy plus 3D semantic, instance, and panoptic occupancy from multi-view images.
Problem
Multi-view pedestrian detection is limited to ground-plane locations, and existing datasets lack voxel-level 3D occupancy and scene semantics for dense pedestrians in large, variable-camera scenes.
Approach
Extends multi-view pedestrian detection to joint 2D ground-plane and 3D voxel occupancy prediction, using non-parametric view transformation (feature projection along rays), a 3D U-Net voxel encoder, and distance-based grouping of pedestrian voxels into instance and panoptic occupancy.
Key results
- Same-scene 2D evaluation on MVP-Occ: OmniOcc averages 96.8 MODA-related F1 of 98.0 on Alley and 96.8 avg MODA vs 90.8 for MVFP (MODA avg 88.0 vs 95.2 F1 ranking reported per table; OmniOcc avg F1 96.8).
- Synthetic-to-real on WildTrack (Facade training): OmniOcc reaches 75.4 MODA and 87.5 F1, versus 52.6 MODA and 74.9 F1 for MVFP.
- Training on all MVP-Occ scenes improves WildTrack transfer to 78.8 MODA and 89.7 F1; CityStreet MODA 57.3 vs 45.6 for MVFP.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OmniOcc instead of MVFP | MVP-Occ same-scene 2D occupancy · MVP-Occ (Avg.) | MODA: 90.8 -> 93.8 | Helped |
| Use OmniOcc instead of MVFP | MVP-Occ same-scene 2D occupancy · MVP-Occ (Avg.) | F1: 95.2 -> 96.8 | Helped |
| Use OmniOcc instead of MVFP | Synthetic-to-real 2D occupancy · WildTrack | F1: 74.9 -> 87.5 | Helped |
| Add weighted cross-entropy loss (ablation) | OmniOcc 3D semantic occupancy · Facade | mIoU: 85.3 -> 89.4 | Helped |
| Add Lovász loss alongside weighted cross-entropy (ablation) | OmniOcc 3D semantic occupancy · Facade | mIoU: 89.4 -> 93.4 | Helped |
| Add affinity loss alongside weighted cross-entropy and Lovász losses (ablation) | OmniOcc 3D semantic occupancy · Facade | mIoU: 93.4 -> 93.7 | Helped |
| Add semantic scene understanding (ablation) | OmniOcc · WildTrack synthetic-to-real 2D occupancy | MODA: 66.5 -> 75.4 | Helped |
| Add semantic scene understanding (ablation) | OmniOcc · WildTrack synthetic-to-real 3D instance occupancy | AP: 38.2 -> 40.2 | Helped |
| Add pedestrian instance grouping (ablation) | OmniOcc · Park same-scene | mIoU: 91.8 -> 91.8 | No clear effect |
| Add pedestrian instance grouping (ablation) | OmniOcc · WildTrack synthetic-to-real | IoU Free: 87.8 -> 87.9 | Helped |
| Train on all MVP-Occ scenes instead of Facade only (ablation) | OmniOcc synthetic-to-real · WildTrack | F1: 87.5 -> 89.7 | Helped |
| Replace ResNet-18 with ResNet-50 (ablation) | OmniOcc · Facade | IoU Ped.: 69.5 -> 70.3 | Helped |
Surprising findings
- Adding more residual blocks to the voxel encoder improves 3D occupancy but lowers 2D pedestrian occupancy performance beyond three blocks.
- Sophisticated location-decoder heads (dilated convolutions, large kernel refiner) gave no improvement over a single convolutional layer.
Limitations
- Synthetic-to-real transfer is weaker for scenes with different camera heights and layouts (e.g., Plaza, Alley, Field).
- MODP is lower than baselines in most same-scene settings due to voxel-level whole-body representation vs ground-plane regression.
- Real WildTrack evaluation uses manually annotated 2D segmentation masks as a proxy for 3D occupancy because no 3D ground truth exists.
Experimental setup
Datasets: MVP-Occ WildTrack CityStreet
Benchmarks: MVP-Occ WildTrack CityStreet
Metrics: MODA MODP F1 mIoU IoU-Ped AP PQ SQ RQ
Baselines: MVDet SHOT GMVD MVFP
Builds on: MVDet MVFP GMVD CARLA
Closest papers by meaning
- CSV-Occ: Fusing Multi-frame Alignment for Occupancy Prediction with Temporal Cross State Space Model and Central Voting Mechanism (ICML 2025 · similarity 0.88)
- Exploiting Continuous Motion Clues for Vision-Based Occupancy Prediction (AAAI 2025 · similarity 0.87)
- PanoOcc: Unified Occupancy Representation for Camera-based 3D Panoptic Segmentation (CVPR 2024 · similarity 0.87)
- OPUS: Occupancy Prediction Using a Sparse Set (NeurIPS 2024 · similarity 0.87)
- OccMamba: Semantic Occupancy Prediction with State Space Models (CVPR 2025 · similarity 0.87)
- OctOcc: High-Resolution 3D Occupancy Prediction with Octree (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.