Research map / Detection and segmentation / 3D detection and driving
M3Net: Multimodal Multi-task Learning for 3D Detection, Segmentation, and Occupancy Prediction in Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes M3Net, a multimodal (LiDAR+camera) multi-task network that jointly performs 3D detection, BEV map segmentation, and 3D occupancy prediction via query-token interaction in a shared decoder, with transformer or Mamba variants.
Problem
Autonomous driving perception usually requires separate models for detection, map segmentation, and occupancy prediction; existing multi-task models underperform single-task models due to gradient conflicts between tasks.
Approach
Modality-adaptive gating (MAFI) fuses LiDAR and camera BEV features; task-specific queries are initialized from BEV features (confidence-selected for detection/segmentation, voxel-based for occupancy); and a task-oriented channel scaling (TCS) module produces per-task channel-scaled BEV features to reduce inter-task conflicts in a shared decoder.
Key results
- M3Net (transformer) achieves 69.0 mAP, 72.4 NDS and 70.4 mean BEV segmentation IoU on nuScenes val, versus 68.5 mAP / 71.4 NDS / 62.7 seg for BEVFusion single-task models.
- M3Net (mamba) reaches 70.8 mean segmentation IoU, outperforming MetaBEV (sep) at 66.9.
- On nuScenes-Occupancy, M3Net (mamba) achieves 24.1 mIoU vs 20.1 for CONet (HR).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use M3Net with a transformer decoder instead of independently trained BEVFusion | BEVFusion · nuScenes val | mAP: +0.5% | Helped |
| Use M3Net with a transformer decoder instead of independently trained BEVFusion | BEVFusion · nuScenes val | NDS: +1.0% | Helped |
| Use M3Net with a transformer decoder instead of independently trained BEVFusion | BEVFusion · nuScenes val | IoU: +7.7% | Helped |
| Use M3Net (transformer) instead of CONet (HR) | CONet (HR) · nuScenes-Occupancy val | mIoU: +3.2% | Helped |
| Use M3Net (mamba) instead of CONet (HR) | CONet (HR) · nuScenes-Occupancy val | mIoU: +4.0% | Helped |
| Add multi-task detection and segmentation training to the M3Net occupancy model (ablation) | M3Net occupancy-only · nuScenes-Occupancy val | mIoU: 6% | Helped |
| Add MAFI to M3Net multi-query (ablation) | M3Net multi-query · nuScenes val | NDS: 0.3% | Helped |
| Add MAFI to M3Net multi-query (ablation) | M3Net multi-query · nuScenes val | IoU: 0.6% | Helped |
| Add MAFI to M3Net multi-query (ablation) | M3Net multi-query · nuScenes-Occupancy val | mIoU: 0.4% | Helped |
| Add TCS to M3Net multi-query + MAFI (ablation) | M3Net multi-query + MAFI · nuScenes val | NDS: 0.7% | Helped |
| Add TCS to M3Net multi-query + MAFI (ablation) | M3Net multi-query + MAFI · nuScenes val | IoU: 0.4% | Helped |
| Add TCS to M3Net multi-query + MAFI (ablation) | M3Net multi-query + MAFI · nuScenes-Occupancy val | mIoU: 0.3% | Helped |
Surprising findings
- Randomly initialized queries performed worse, motivating BEV-based query initialization.
- Without TCS and MAFI, the multi-query baseline loses about 1.4 points (NDS+IoU) versus a larger 9.4-point drop for BEVFusion multi-task.
Limitations
- Multi-task gains are reported mainly against BEVFusion and MetaBEV; comparisons depend on the specific training schedule used.
- Occupancy prediction is run on a low-resolution 180x180x5 feature space, upsampled to output resolution.
Experimental setup
Datasets: nuScenes OpenOccupancy
Benchmarks: nuScenes val nuScenes-Occupancy val
Metrics: mAP NDS mIoU IoU
Baselines: BEVFusion BEVFusion-MTL MetaBEV FULLER CONet MonoScene TPVFormer CenterPoint
Builds on: BEVFusion Mask2Former VMamba OpenOccupancy MetaBEV
Closest papers by meaning
- MMTL-UniAD: A Unified Framework for Multimodal and Multi-Task Learning in Assistive Driving Perception (CVPR 2025 · similarity 0.87)
- H-MBA: Hierarchical MamBa Adaptation for Multi-Modal Video Understanding in Autonomous Driving (AAAI 2025 · similarity 0.87)
- Holistic Autonomous Driving Understanding by Bird's-Eye-View Injected Multi-Modal Large Models (CVPR 2024 · similarity 0.85)
- Achieving Speed-Accuracy Balance in Vision-based 3D Occupancy Prediction via Geometric-Semantic Disentanglement (AAAI 2025 · similarity 0.85)
- MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI (ICML 2024 · similarity 0.84)
- S4-Driver: Scalable Self-Supervised Driving Multimodal Large Language Model with Spatio-Temporal Visual Representation (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.