Research map / Detection and segmentation / 3D detection and driving
3DiffTection: 3D Object Detection with Geometry-Aware Diffusion Features
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts pretrained Stable Diffusion into a 3D-aware feature extractor for single-image 3D object detection by training a geometric ControlNet (view synthesis with epipolar warp) and a semantic ControlNet alongside a Cube R-CNN-style detection head, with virtual-view NMS ensembling.
Problem
3D object detection from a single image is limited by scarce annotated 3D data, and diffusion features pretrained on 2D image-text pairs lack 3D awareness (point correspondences are confused by repeated patterns).
Approach
Inject 3D awareness into frozen Stable Diffusion features by training a ControlNet for novel-view synthesis with an epipolar warp operator on posed video frame pairs (no manual labels), then add a second trainable semantic ControlNet co-trained with a 3D box head and ensemble predictions over virtual views.
Key results
- On Omni3D-ARKitScenes (512x512), 3DiffTection reaches 43.75 AP3D vs 34.32 for CubeRCNN-DLA and 41.72 for CubeRCNN-DLA-Aug (trained on ~6x more data).
- Zero-shot transfer from ARKitScenes to Omni3D-SUNRGBD gives 27.81 AP3D (w/o 2D GT) with semantic ControlNet, vs 22.72 for CubeRCNN fine-tuned.
- With geometric ControlNet only (frozen backbone), AP3D improves from 28.86 (DIFT) to 31.20 with two-view NVS training; full model with NV-ensemble reaches 39.22 at 256x256.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use 3DiffTection instead of DreamTchr-Res50 | Omni3D-ARKitScenes 3D detection at 256×256 · Omni3D-ARKitScenes test | AP3D: 33.20 -> 39.22 | Helped |
| use 3DiffTection instead of CubeRCNN-DLA-Aug | Omni3D-ARKitScenes 3D detection at 512×512 · Omni3D-ARKitScenes test | AP3D: 41.72 -> 43.75 | Helped |
| add semantic ControlNet (ablation) | 3DiffTection on Omni3D-SUNRGBD with tuning · Omni3D-SUNRGBD, tuned | AP3D: 23.11 -> 27.81 | Helped |
| add semantic ControlNet (ablation) | 3DiffTection on Omni3D-SUNRGBD in zero-shot evaluation without 2D GT · Omni3D-SUNRGBD, zero-shot without 2D GT | AP3D: 17.37 -> 22.64 | Helped |
| add semantic ControlNet (ablation) | 3DiffTection on Omni3D-SUNRGBD in zero-shot evaluation with 2D GT · Omni3D-SUNRGBD, zero-shot with 2D GT | AP3D: 26.94 -> 30.14 | Helped |
| use geometric ControlNet trained with 2 NVS views instead of single-view denoising (ablation) | Stable Diffusion features with a 3D detection head · Omni3D-ARKitScenes test | AP3D: 26.05 -> 31.20 | Helped |
| use geometric ControlNet trained with 2 NVS views instead of vanilla Stable Diffusion features (ablation) | Stable Diffusion features with a 3D detection head · Omni3D-ARKitScenes test | AP3D: 28.86 -> 31.20 | Helped |
| add semantic ControlNet to the geometric ControlNet model (ablation) | 3DiffTection with 3D detection head · Omni3D-ARKitScenes test | AP3D: 31.20 -> 38.72 | Helped |
| add novel-view ensemble (ablation) | 3DiffTection · Omni3D-ARKitScenes test | AP3D: 38.72 -> 39.22 | Helped |
| use Stable Diffusion features instead of VIT-B (MAE) features | 3D detection head with frozen backbone · Omni3D-ARKitScenes test | AP3D: 25.23 -> 28.86 | Helped |
| use 3DiffTection with 50% of labels instead of prior methods trained with 100% of labels | Omni3D-ARKitScenes · Omni3D-ARKitScenes | AP3D-N: 2.28 | Helped |
| use geometric ControlNet features instead of DIFT-SD features | 3D detection on Omni3D-SUNRGBD · Omni3D-SUNRGBD, tuned | AP3D: 21.92 -> 23.11 | Helped |
Surprising findings
- Restricting NVS training to single-view (denoising) data lowers AP3D by 2.81 points versus two-view epipolar NVS training, indicating the gain comes from view synthesis rather than extra denoising training.
Limitations
- Requires image pairs with accurate camera poses.
- Struggles with dynamic objects in in-the-wild videos.
- Stable Diffusion backbone is memory and runtime heavy (about 7.5 fps on a 3090Ti).
Experimental setup
Datasets: ARKitScenes Omni3D-ARKitScenes Omni3D-SUNRGBD ScanNet
Benchmarks: Omni3D-ARKitScenes Omni3D-SUNRGBD ScanNet
Metrics: AP3D AP3D@15 AP3D@25 AP3D@50 AP2D
Baselines: CubeRCNN-DLA CubeRCNN-DLA-Aug DreamTeacher-Res50 NeRF-Det-R50 ImVoxelNet DIFT CubeRCNN-ViT-B (MAE)
Builds on: Stable Diffusion ControlNet Cube R-CNN DIFT SparseFusion
Closest papers by meaning
- MonoDiff: Monocular 3D Object Detection and Pose Estimation with Diffusion Models (CVPR 2024 · similarity 0.88)
- DiffuBox: Refining 3D Object Detection with Point Diffusion (NeurIPS 2024 · similarity 0.87)
- DriveGEN: Generalized and Robust 3D Detection in Driving via Controllable Text-to-Image Diffusion Generation (CVPR 2025 · similarity 0.87)
- SP3D: Boosting Sparsely-Supervised 3D Object Detection via Accurate Cross-Modal Semantic Prompts (CVPR 2025 · similarity 0.87)
- ReDiffDet: Rotation-equivariant Diffusion Model for Oriented Object Detection (CVPR 2025 · similarity 0.86)
- Dual-Perspective Knowledge Enrichment for Semi-supervised 3D Object Detection (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.