Research map / Detection and segmentation / 3D detection and driving
VSRD: Instance-Aware Volumetric Silhouette Rendering for Weakly Supervised 3D Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VSRD, a weakly supervised monocular 3D object detection framework that auto-labels 3D boxes from multi-view 2D instance masks and 2D boxes via instance-aware SDF silhouette rendering, then trains monocular detectors on confidence-weighted pseudo labels.
Problem
Monocular 3D object detection normally requires expensive LiDAR-based 3D box annotations; the paper asks whether accurate 3D detectors can be trained using only 2D supervision (instance masks and 2D boxes).
Approach
Represent each instance as an SDF composed of a cuboid SDF plus a learned residual distance field (hypernetwork-generated), and render instance masks by integrating softmin-averaged instance labels along rays with NeuS-style volume weights, so 3D boxes are optimized end-to-end against 2D masks; per-pseudo-label confidence from multi-view projection IoU weights detector regression loss.
Key results
- On KITTI-360 test, VSRD with S-WeakM3D reaches 51.09/42.94 AP_BEV/AP_3D@0.3 (Easy) vs 29.89/21.25 for WeakM3D; VSRD with MonoDETR reaches 58.40/50.86 at the same setting.
- Auto-labeling on KITTI-360 train with confidence>=0.8 gives 84.54/80.25 AP_BEV/AP_3D@0.3 (Easy) vs 75.56/17.18 for Autolabels.
- Semi-supervised: detector pre-trained on KITTI-360 pseudo labels and fine-tuned on 50% of KITTI labels significantly outperforms training from scratch on full KITTI (MonoDETR 37.99/29.36 AP_BEV/AP_3D@0.7 Easy).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train MonoDETR with VSRD pseudo labels instead of full 3D supervision | MonoDETR · KITTI-360 test, AP3D@0.5 | AP3D@0.5: 41.01 -> 21.77 (Easy); 30.38 -> 16.46 (Hard) | Hurt |
| Train MonoFlex with VSRD pseudo labels instead of full 3D supervision | MonoFlex · KITTI-360 test, AP3D@0.5 | AP3D@0.5: 43.11 -> 18.53 (Easy); 34.43 -> 13.60 (Hard) | Hurt |
| Replace WeakM3D supervision with VSRD pseudo labels | S-WeakM3D · KITTI-360 test, AP3D@0.5 | AP3D@0.5: 2.96 -> 11.91 (Easy); 2.01 -> 8.46 (Hard) | Helped |
| Pre-train MonoDETR with VSRD on KITTI-360, then fine-tune with 50% labeled KITTI data | MonoDETR · KITTI validation, AP3D@0.7 | AP3D@0.7: 21.48/17.93 vs 20.64/17.30 | Helped |
| Add silhouette loss to projection-loss auto-labeling (ablation) | VSRD multi-view auto-labeling · KITTI-360 training, Easy AP3D@0.5 | AP3D@0.5: 23.33 -> 26.84 | Helped |
| Add silhouette loss to projection-loss auto-labeling (ablation) | VSRD multi-view auto-labeling · KITTI-360 training, Hard AP3D@0.5 | AP3D@0.5: 24.82 -> 25.58 | Helped |
| Add residual distance field (RDF) to projection and silhouette losses (ablation) | VSRD multi-view auto-labeling · KITTI-360 training, Easy AP3D@0.5 | AP3D@0.5: 26.84 -> 31.11 | Helped |
| Add residual distance field (RDF) to projection and silhouette losses (ablation) | VSRD multi-view auto-labeling · KITTI-360 training, Hard AP3D@0.5 | AP3D@0.5: 25.58 -> 30.16 | Helped |
| Add confidence-based regression loss weighting (ablation) | S-WeakM3D trained with VSRD pseudo labels · KITTI-360 test, Easy AP3D@0.5 | AP3D@0.5: 8.25 -> 11.91 | Helped |
| Add confidence-based regression loss weighting (ablation) | S-WeakM3D trained with VSRD pseudo labels · KITTI-360 test, Hard AP3D@0.5 | AP3D@0.5: 4.41 -> 8.46 | Helped |
Surprising findings
- Zero-shot transfer from KITTI-360 pseudo labels to KITTI is very poor (AP_3D@0.7 around 0.002 Easy), attributed to camera-parameter sensitivity of monocular depth estimation.
- Without confidence weighting, auto-labeling pseudo labels give high BEV AP but low AP_3D with Autolabels; the residual distance field gave the largest gain in the ablation.
Limitations
- Pseudo labels are unreliable for dynamic, occluded, or truncated objects, requiring confidence weighting.
- Zero-shot cross-dataset transfer to KITTI performs very poorly due to camera parameter differences.
- Evaluated only on the Car category of KITTI-360 with a modified two-level difficulty protocol.
Experimental setup
Datasets: KITTI-360 KITTI
Benchmarks: KITTI-360 KITTI
Metrics: AP_BEV@0.3 AP_3D@0.3 AP_BEV@0.5 AP_3D@0.5
Baselines: WeakM3D Autolabels MonoFlex MonoDETR S-WeakM3D
Builds on: NeuS NeRF WeakM3D Autolabels MonoDETR
Closest papers by meaning
- WeakPCSOD: Overcoming the Bias of Box Annotations for Weakly Supervised Point Cloud Salient Object Detection (AAAI 2024 · similarity 0.88)
- Weak-to-Strong 3D Object Detection with X-Ray Distillation (CVPR 2024 · similarity 0.88)
- LabelAny3D: Label Any Object 3D in the Wild (NeurIPS 2025 · similarity 0.88)
- Weakly Supervised Monocular 3D Detection with a Single-View Image (CVPR 2024 · similarity 0.87)
- MonoMAE: Enhancing Monocular 3D Detection through Depth-Aware Masked Autoencoders (NeurIPS 2024 · similarity 0.87)
- Beyond Virtual Points: Depth-Enhanced LiDAR-only 3D Object Detection with Semi-Supervised Learning (Student Abstract) (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.