Research map / Detection and segmentation / 3D detection and driving
Leveraging Temporal Cues for Semi-Supervised Multi-View 3D Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Semi-supervised framework for temporal camera-only 3D detection that trains a StreamPETR detector on forward and backward RGB sequences, ensembles their pseudo-labels, refines them with 2D-3D Hungarian matching and tracking, and adds an object-query conditioned masked reconstruction loss.
Problem
Camera-only 3D object detection requires many human-annotated frames, and standard pseudo-labeling underperforms in the temporal camera-only setting because pseudo-label 3D localization is poor.
Approach
Train one temporal detector on both forward and reversed sequences and ensemble both directions' pseudo-labels (front from backward pass, behind from forward pass), filter with 2D-3D Hungarian matching between the model's own 2D and 3D heads instead of confidence thresholds, infill with tracking, and condition masked image reconstruction on object queries so the reconstruction task complements det
Key results
- On nuPlan with 800 labeled frames, the method reaches 0.222 mAP and 0.280 NDS versus 0.187 mAP for confidence pseudo-labeling and 0.151 for labeled-only.
- On nuScenes with 800 labeled frames, the method reaches 0.137 mAP versus 0.116 for UniPAD + pseudo-labeling and 0.103 for labeled-only.
- Forward-backward ensembling substantially reduces mATE (0.916 to 0.855 in the ablation) in addition to improving mAP and NDS.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the full semi-supervised framework instead of labeled-only training | StreamPETR · nuPlan, 800 labeled frames | mAP: 0.151 -> 0.222 | Helped |
| use the full semi-supervised framework instead of confidence pseudo-labeling | StreamPETR · nuPlan, 2000 labeled frames | mAP: 0.269 -> 0.271 | Helped |
| use the full semi-supervised framework instead of confidence pseudo-labeling | StreamPETR · nuPlan, 4000 labeled frames | mAP: 0.304 -> 0.331 | Helped |
| use the full semi-supervised framework instead of UniPAD plus pseudo-labeling | StreamPETR · nuScenes, 800 labeled frames | mAP: 0.116 -> 0.137 | Helped |
| add pseudo-labeling (ablation) | StreamPETR · nuPlan, 800 labeled and 60k unlabeled samples | mAP: 0.155 -> 0.182 | Helped |
| add forward-backward ensembling (ablation) | StreamPETR with pseudo-labeling · nuPlan, 800 labeled and 60k unlabeled samples | mATE: 0.924 -> 0.855 | Helped |
| add independent masked reconstruction (ablation) | StreamPETR with forward-backward ensembling · nuPlan, 800 labeled and 60k unlabeled samples | mATE: 0.855 -> 0.862 | Hurt |
| condition masked reconstruction on object queries (ablation) | StreamPETR with masked reconstruction · nuPlan, 800 labeled and 60k unlabeled samples | mATE: 0.862 -> 0.847 | Helped |
| add 3D tracking and label propagation (ablation) | StreamPETR with query-conditioned masked reconstruction · nuPlan, 800 labeled and 60k unlabeled samples | mAP: 0.209 -> 0.215 | Helped |
| replace confidence thresholding with 2D-3D Hungarian matching (ablation) | StreamPETR with 3D tracking · nuPlan, 800 labeled and 60k unlabeled samples | mAP: 0.215 -> 0.219 | Helped |
Surprising findings
- Adding a masked reconstruction head directly on the backbone hurt detection performance, while conditioning reconstruction on object queries helped.
- Using 2D and 3D detections from the same model via Hungarian matching improved pseudo-label quality, even though they come from one model and modality.
Limitations
- Relies on a single base detector (StreamPETR) and ConvNeXt-S backbone; generality to other detectors not shown in the text.
- Pseudo-labeling still yields poor 3D localization for some objects, as mATE remains high in several settings.
Experimental setup
Datasets: nuScenes nuPlan
Benchmarks: nuScenes nuPlan
Metrics: mAP NDS mATE mASE mAOE mAVE mAAE
Baselines: Labeled-Only Confidence Pseudo-Labeling UniPAD + Pseudo-Labeling UniPAD
Builds on: StreamPETR DetMatch SparK UniPAD DPL
Closest papers by meaning
- Decoupled Pseudo-labeling for Semi-Supervised Monocular 3D Object Detection (CVPR 2024 · similarity 0.89)
- Dual-Perspective Knowledge Enrichment for Semi-supervised 3D Object Detection (AAAI 2024 · similarity 0.89)
- Leveraging Imagery Data with Spatial Point Prior for Weakly Semi-supervised 3D Object Detection (AAAI 2024 · similarity 0.88)
- Semi-supervised 3D Object Detection with PatchTeacher and PillarMix (AAAI 2024 · similarity 0.88)
- SP3D: Boosting Sparsely-Supervised 3D Object Detection via Accurate Cross-Modal Semantic Prompts (CVPR 2025 · similarity 0.87)
- LabelAny3D: Label Any Object 3D in the Wild (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.