Research map / Detection and segmentation / 3D detection and driving
Collaborative Semantic Occupancy Prediction with Hybrid Feature Fusion in Connected Automated Vehicles
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoHFF, the first camera-based collaborative 3D semantic occupancy prediction framework, sharing orthogonal-plane features between connected vehicles and fusing occupancy and semantic task features, and extends OPV2V with 12-class voxel semantic labels.
Problem
Camera-based collaborative perception has only used 3D bounding boxes or BEV representations, lacking comprehensive 3D semantic occupancy prediction under V2X communication bandwidth limits.
Approach
Pretrain separate occupancy and semantic segmentation task networks, share compressed orthogonal-plane (xz, yz) semantic features between CAVs via learnable masks and deformable plane self-attention, and fuse task features with a 3D depth-wise conv network to produce collaborative semantic voxel predictions.
Key results
- Collaboration improves overall semantic occupancy IoU by about 31% and mIoU by about 37% over single-vehicle prediction (Tab. 4).
- With 2 agents, CoHFF reaches AP@0.7 of 27.95 vs 12.60 for CoCa3D-2 on vehicle 3D detection (Tab. 1).
- Sparsifying shared plane features by 0.99 cuts communication volume from 16.53 MB to 0.17 MB, with IoU dropping from 50.46 to 48.02 (Tab. 3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| enable collaboration across CAVs (ablation) | CoHFF semantic occupancy prediction · OPV2V semantic occupancy prediction | IoU: 38.52 -> 50.46 | Helped |
| enable collaboration across CAVs (ablation) | CoHFF semantic occupancy prediction · OPV2V semantic occupancy prediction | mIoU: 24.85 -> 34.16 | Helped |
| add semantic segmentation task features to occupancy prediction (ablation) | CoHFF occupancy prediction · OPV2V occupancy prediction | IoU: 67.22 -> 76.62 | Helped |
| add task feature fusion to occupancy prediction (ablation) | CoHFF occupancy prediction · OPV2V occupancy prediction | IoU: 76.62 -> 86.37 | Helped |
| add occupancy prediction features to semantic segmentation (ablation) | CoHFF semantic segmentation · OPV2V semantic segmentation | IoU: 41.30 -> 42.11 | Helped |
| add task feature fusion to semantic segmentation (ablation) | CoHFF semantic segmentation · OPV2V semantic segmentation | IoU: 42.11 -> 51.38 | Helped |
| increase feature sparsification rate from 0.00 to 0.50 (ablation) | CoHFF · OPV2V semantic occupancy prediction | IoU: 50.46 -> 49.56 | Hurt |
| increase feature sparsification rate from 0.00 to 0.80 (ablation) | CoHFF · OPV2V semantic occupancy prediction | IoU: 50.46 -> 49.53 | Hurt |
| increase feature sparsification rate from 0.00 to 0.95 (ablation) | CoHFF · OPV2V semantic occupancy prediction | mIoU: 34.16 -> 30.13 | Hurt |
| increase feature sparsification rate from 0.00 to 0.99 (ablation) | CoHFF · OPV2V semantic occupancy prediction | IoU: 50.46 -> 48.02 | Hurt |
Surprising findings
- Adding semantic features to occupancy prediction raises IoU but lowers mIoU, attributed to the model favoring easily detected classes.
Limitations
- Evaluated only on simulation data (CARLA/OPV2V); real-world performance unverified.
- Occupancy and semantic tasks are pretrained separately, with the overall pipeline depending on multiple stages.
- Comparisons with baselines are on different agent counts and downstream tasks, not directly on occupancy.
Experimental setup
Datasets: OPV2V OPV2V+
Benchmarks: OPV2V (extended with semantic occupancy labels)
Metrics: IoU mIoU AP@0.5 AP@0.7 BEV IoU communication volume (MB)
Baselines: CoBEVT CoCa3D DiscoNet V2X-ViT Where2Comm
Builds on: TPVFormer K-Planes CaDNN Where2Comm CoBEVT
Closest papers by meaning
- Achieving Speed-Accuracy Balance in Vision-based 3D Occupancy Prediction via Geometric-Semantic Disentanglement (AAAI 2025 · similarity 0.91)
- CSV-Occ: Fusing Multi-frame Alignment for Occupancy Prediction with Temporal Cross State Space Model and Central Voting Mechanism (ICML 2025 · similarity 0.90)
- COTR: Compact Occupancy TRansformer for Vision-based 3D Occupancy Prediction (CVPR 2024 · similarity 0.89)
- OctOcc: High-Resolution 3D Occupancy Prediction with Octree (AAAI 2024 · similarity 0.89)
- EvOcc: Accurate Semantic Occupancy for Automated Driving Using Evidence Theory (CVPR 2025 · similarity 0.88)
- SDGOCC: Semantic and Depth-Guided Bird's-Eye View Transformation for 3D Multimodal Occupancy Prediction (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.