Research map / Detection and segmentation / Weakly supervised detection and segmentation
MWSIS: Multimodal Weakly Supervised Instance Segmentation with 2D Box Annotations for Autonomous Driving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MWSIS, which trains both 2D image and 3D LiDAR instance segmentors simultaneously using only 2D box annotations, with label-correction modules (IPG, SPG, PVC, RSC) and cross-modal response distillation (CSCS).
Problem
Instance segmentation for autonomous driving requires costly mask annotations; prior weak methods use 2D or 3D boxes for a single modality, and none trains 2D and 3D segmentors simultaneously from only 2D box labels, which yields noisy pseudo labels.
Approach
Use 2D boxes as the only annotation, generate and correct pseudo masks in each modality (IPG for images; depth-clustering ring segments with SPG, PVC voting and RSC correction for point clouds), and couple the two student-teacher branches through consistency sparse cross-modal distillation (CSCS).
Key results
- 2D instance segmentation with 2D box supervision reaches 37.20 mAP vs 34.61 for BoxInst (+2.59), at 24 epochs on Waymo val.
- 3D instance segmentation with 2D box supervision reaches 46.93 mAP vs 34.18 baseline (+12.75), outperforming SparseUNet fully supervised 3D instance segmentation after pretraining with 5% of full annotations.
- Using the 2D-box-trained 3D backbone to pretrain FSD improves Waymo val mAPH L2 from 69.7 to 70.2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MWSIS instead of BoxInst | 2D instance segmentation · Waymo val | mAP AP: 34.61 -> 37.20; 2.59% mAP improvement | Helped |
| Use MWSIS with 2D boxes instead of the clustered 2D-box baseline | 3D instance segmentation · Waymo val | mAP AP: 34.18 -> 46.93; 12.75% mAP | Helped |
| Pre-train the FSD backbone with MWSIS | FSD · Waymo val 3D detection | mAP/mAPH L1: 78.7/76.3 -> 79.2/76.8 | Helped |
| Add SPG using point-cloud spatial priors (ablation) | SparseUNet with CCL, 2D-box weak supervision · Waymo val 3D instance segmentation | mAP: 34.18 -> 42.32; 8.14% mAP | Helped |
| Add PVC historical prediction voting (ablation) | SparseUNet with CCL and SPG, 2D-box weak supervision · Waymo val 3D instance segmentation | mAP: 42.32 -> 44.71; 2.32% mAP | Helped |
| Add RSC ring-segment label correction (ablation) | SparseUNet with CCL, SPG, and PVC, 2D-box weak supervision · Waymo val 3D instance segmentation | mAP: 44.71 -> 45.76; 1.05% mAP | Helped |
| Add CSCS cross-modal distillation (ablation) | MWSIS 3D segmentor · Waymo val 3D instance segmentation | mAP: 45.76 -> 46.93; 1.17% mAP | Helped |
| Add CSCS cross-modal distillation (ablation) | BoxInst 2D segmentor · Waymo val 2D instance segmentation | mAP: 36.77 -> 37.20; 0.43% mAP | Helped |
| Add IoU-based mask weighting in IPG (ablation) | BoxInst · Waymo val 2D instance segmentation | mAP: 34.61 -> 35.72; 1.11% mAP | Helped |
| Add prediction-score weighting to IPG IoU weighting (ablation) | BoxInst · Waymo val 2D instance segmentation | mAP: 34.61 -> 36.49; 2.16% mAP | Helped |
| Set IPG k=5 instead of k=1 (ablation) | BoxInst · Waymo val 2D instance segmentation | mAP: 36.77 -> 35.62 | Hurt |
| Use 4 historical votes in PVC rather than 0 (ablation) | SparseUNet with SPG · Waymo val 3D instance segmentation | mAP: 42.32 -> 44.71; 2.12% mAP | Helped |
Surprising findings
- Weak 2D-box-only 3D instance segmentation (46.93 mAP) surpasses the 3D-box-supervised variant with SAM-free pipeline reported in the same table only at certain settings; the 2D-box variant also slightly beats the fully supervised cyclist AP in 2D.
Limitations
- Evaluated only on Waymo; no results on nuScenes or other datasets.
- Relies on hand-tuned thresholds and dynamic depth/window parameters for the DCS and RSC algorithms.
- Gains on 2D instance segmentation are modest relative to LWSIS (37.20 vs 37.77 mAP), which uses 3D boxes.
Experimental setup
Datasets: Waymo Open Dataset
Benchmarks: Waymo Open Dataset validation set
Metrics: mAP AP50 AP75 mIoU
Baselines: BoxInst CondInst LWSIS Box2Mask SAM-based pseudo masks CPC CPS
Builds on: BoxInst CondInst SparseUNet CPS mean teacher
Closest papers by meaning
- WISH: Weakly Supervised Instance Segmentation using Heterogeneous Labels (CVPR 2025 · similarity 0.88)
- Weakly Supervised Semantic Segmentation for Driving Scenes (AAAI 2024 · similarity 0.88)
- Extreme Point Supervised Instance Segmentation (CVPR 2024 · similarity 0.87)
- Sketchy Bounding-box Supervision for 3D Instance Segmentation (CVPR 2025 · similarity 0.87)
- P2Seg: Pointly-supervised Segmentation via Mutual Distillation (ICLR 2024 · similarity 0.86)
- 3D Annotation-Free Learning by Distilling 2D Open-Vocabulary Segmentation Models for Autonomous Driving (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.