Research map / Detection and segmentation / 3D detection and driving
RayFusion: Ray Fusion Enhanced Collaborative Visual Perception
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RayFusion, a camera-only collaborative 3D object detector that shares per-instance ray occupancy (from depth distributions) and fuses it with multi-scale pyramid window attention to reduce false positives along camera rays.
Problem
Camera-only collaborative 3D object detection in autonomous driving suffers from depth estimation ambiguity, causing multiple false positives along each camera ray; existing collaborative methods do not explicitly model multi-view camera imaging of the same object.
Approach
Encode each instance's camera rays (optical center and direction) together with the depth-bin occupancy distribution shared by collaborators, fuse across cameras with attention, and aggregate instances with a pyramid of receptive-field-restricted window self-attention, using instance-level sharing with spatial-temporal alignment.
Key results
- RayFusion reaches 11.54 AP70 on DAIR-V2X, 53.14 AP70 on V2XSet, and 74.25 AP70 on OPV2V, improving over IFTR by 3.64, 3.47, and 8.21 AP70 respectively.
- On OPV2V, the spatial-temporal alignment module contributes a 28.99 AP70 gain in the ablation, the multi-scale aggregation 4.61, and ray occupancy encoding 2.21.
- Communication cost is about 276.6 KB per frame per agent for RayFusion versus 18.8 MB for IFTR under the OPV2V setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add spatial-temporal alignment (STA) (ablation) | RayFusion base model · OPV2V | AP70: +28.99 | Helped |
| Add spatial-temporal alignment (STA) (ablation) | RayFusion base model · DAIR-V2X | AP70: +4.19 | Helped |
| Add multi-scale instance feature aggregation (MIFA) (ablation) | RayFusion base model · OPV2V | AP70: +4.61 | Helped |
| Add multi-scale instance feature aggregation (MIFA) (ablation) | RayFusion base model · DAIR-V2X | AP70: +1.03 | Helped |
| Add ray occupancy encoding (ROE) (ablation) | RayFusion base model · OPV2V | AP70: +2.21 | Helped |
| Add ray occupancy encoding (ROE) (ablation) | RayFusion base model · DAIR-V2X | AP70: +1.34 | Helped |
| Replace pyramid window self-attention with vanilla multi-head self-attention (ablation) | RayFusion MIFA · OPV2V and DAIR-V2X | AP70 | Hurt |
| Add optical-center and ray-direction encoding (ablation) | RayFusion ROE · collaborative 3D detection | AP | Helped |
| Add occupancy information encoding (ablation) | RayFusion ROE · collaborative 3D detection | AP | Helped |
| Remove high-dimensional mapping from ray encoding (ablation) | RayFusion ROE · collaborative 3D detection | AP | Hurt |
| Add ray occupancy information encoding (ablation) | RayFusion · OPV2V | precision and recall | Helped |
Limitations
- Relies on known agent poses for coordinate transformation; collaboration under unknown poses is not addressed.
- Robustness to communication delay is limited; authors propose better occupancy representations for future occupancy prediction.
Experimental setup
Datasets: DAIR-V2X V2XSet OPV2V
Benchmarks: DAIR-V2X V2XSet OPV2V
Metrics: AP50 AP70
Baselines: No Collaboration Late Fusion V2VNet V2X-ViT Where2comm CoBEVT CoAlign IFTR CoSparse4D
Builds on: Sparse4D CoSparse4D IFTR BEVFormer Fourier positional encoding
Closest papers by meaning
- AGC-Drive: A Large-Scale Dataset for Real-World Aerial-Ground Collaboration in Driving Scenarios (NeurIPS 2025 · similarity 0.85)
- CoSDH: Communication-Efficient Collaborative Perception via Supply-Demand Awareness and Intermediate-Late Hybridization (CVPR 2025 · similarity 0.84)
- RCP-Bench: Benchmarking Robustness for Collaborative Perception Under Diverse Corruptions (CVPR 2025 · similarity 0.84)
- Privacy-Preserving V2X Collaborative Perception Integrating Unknown Collaborators (AAAI 2025 · similarity 0.84)
- IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detection (CVPR 2024 · similarity 0.83)
- FRUC: Feedforward Dynamic Scene Reconstruction from Uncalibrated Collaborative Driving Views (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.