Research map / Detection and segmentation / 3D detection and driving
IS-Fusion: Instance-Scene Collaborative Fusion for Multimodal 3D Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IS-Fusion, a LiDAR-camera BEV fusion framework that jointly models instance-level and scene-level multimodal context via hierarchical scene fusion and instance-guided fusion for 3D object detection on nuScenes.
Problem
Multimodal 3D object detection in BEV representations, where small, sparse foreground objects are overwhelmed by background during scene-level fusion.
Approach
Combines hierarchical scene fusion (Point-to-Grid and Grid-to-Region transformers) with an instance-guided module that selects top-K instance candidates, models inter-instance relations, aggregates local image/point context via deformable attention, and feeds instances back to the BEV scene through instance-to-scene cross-attention.
Key results
- On the nuScenes validation set, IS-Fusion reaches 72.8% mAP and 74.0% NDS, outperforming CMT (70.3% mAP) and SparseFusion (71.0% mAP).
- On the nuScenes test set, IS-Fusion achieves 73.0% mAP and 75.2% NDS; with test-time augmentation and model ensemble (IS-Fusion†) it reaches 76.5% mAP.
- Ablation: adding IGF to Baseline-LC improves mAP from 69.4% to 70.9%, while the full HSF+IGF model reaches 72.8%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add HSF to simple multimodal fusion (ablation) | IS-FUSION · nuScenes validation | mAP: 69.4 -> 71.6 | Helped |
| Add IGF to simple multimodal fusion (ablation) | IS-FUSION · nuScenes validation | mAP: 69.4 -> 70.9 | Helped |
| Add both HSF and IGF (ablation) | IS-FUSION · nuScenes validation | mAP: 69.4 -> 72.8 | Helped |
| Add Point-to-Grid attention (ablation) | HSF · nuScenes validation | mAP: 69.4 -> 69.9 | Helped |
| Add Grid-to-Region attention (ablation) | HSF · nuScenes validation | mAP: 69.4 -> 71.2 | Helped |
| Use full HSF with both attention components (ablation) | IS-FUSION · nuScenes validation | mAP: 69.4 -> 71.6 | Helped |
| Use ResNet-50 instead of Swin-T as image encoder (ablation) | IS-FUSION · nuScenes validation | mAP: 71.3 vs 72.8 | Hurt |
| Use CSPNet instead of Swin-T as image encoder (ablation) | IS-FUSION · nuScenes validation | mAP: 71.7 vs 72.8 | Hurt |
| Reduce Swin-T input resolution to 256 × 704 (ablation) | IS-FUSION · nuScenes validation | mAP: 72.4 vs 72.8 | Hurt |
| Set instance count K=64 instead of K=200 (ablation) | IGF · nuScenes validation | mAP: 69.8 vs 70.9 | Hurt |
| Set sampled neighbor count D=32 instead of D=16 (ablation) | IGF · nuScenes validation | mAP: 70.6 vs 70.9 | Hurt |
| Set instance count K=300 instead of K=200 (ablation) | IGF · nuScenes validation | mAP: 70.4 vs 70.9 | Hurt |
Limitations
- Full model is slower than some concurrent methods (3.2 FPS vs. SparseFusion 5.3 FPS).
- Gains from test-time augmentation and ensembling are needed to reach the top leaderboard number.
Experimental setup
Datasets: nuScenes nuImages
Benchmarks: nuScenes test set nuScenes validation set
Metrics: mAP NDS FPS
Baselines: TransFusion-L BEVFusion DeepInteraction CMT SparseFusion FocalFormer3D MSMDFusion UniTR ObjectFusion AutoAlignV2
Builds on: TransFusion BEVFusion CenterPoint Swin Transformer VoxelNet
Closest papers by meaning
- Instance-Aware Multi-Camera 3D Object Detection with Structural Priors Mining and Self-Boosting Learning (AAAI 2024 · similarity 0.87)
- CRT-Fusion: Camera, Radar, Temporal Fusion Using Motion Information for 3D Object Detection (NeurIPS 2024 · similarity 0.87)
- RaCFormer: Towards High-Quality 3D Object Detection via Query-based Radar-Camera Fusion (CVPR 2025 · similarity 0.86)
- E2E-MFD: Towards End-to-End Synchronous Multimodal Fusion Detection (NeurIPS 2024 · similarity 0.86)
- CorrBEV: Multi-View 3D Object Detection by Correlation Learning with Multi-modal Prototypes (CVPR 2025 · similarity 0.86)
- Towards Generalizable Multi-Camera 3D Object Detection via Perspective Rendering (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.