Research map / Detection and segmentation / 3D detection and driving

RefineAny3D: Depth Refinement as Semantic Alignment for Monocular 3D Detection

NeurIPS 2026 · Detection and segmentation › 3D detection and driving · approach: Monocular 3D object detection · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes RefineAny3D, a VLM-based post-hoc depth refinement module for monocular 3D detection that emits discrete direction and magnitude action tokens judged from wireframe-projected boxes, improving closed-set, open-vocabulary, and auto-labeling pipelines.

Problem

Object-level depth from depth foundation models and detectors is too imprecise for tight 3D IoU thresholds in monocular 3D detection, degrading accuracy across closed-set, open-vocabulary, and auto-labeling pipelines.

Approach

Reformulate depth refinement as a 2D semantic alignment judgment: render the projected 3D box as a wireframe overlay and let a VLM choose discrete direction and magnitude action tokens (with CoT), applied iteratively, instead of regressing a metric depth value.

Key results

• On Refine3D Standard split, RefineAny3D reaches 89.2 DirAcc and 76.7 FullAcc versus 28.7/11.4 for off-the-shelf Qwen3-VL-8B.
• Applied to MonoCoP on KITTI val, AP3D at IoU 0.7 rises from 32.06 to 35.62 (Easy) and 23.98 to 27.47 (Moderate).
• On top of DetAny3D with GT 2D boxes, Omni3D AP3D rises from 34.38 to 38.73 (+4.35).

Surprising findings

• Substituting MoGe2 depth for Cube R-CNN predicted depth lowers AP3D by 3.68 (36.00 to 32.54), while oracle depth gives 65.92.
• Geometric projection fitting and a DINOv2 action classifier both degrade MonoCoP performance, while the VLM-based refiner improves it.

Limitations

• Refinement relies on a VLM at inference, adding per-object query cost via iterative multi-step decoding.
• Improvement on Hard split of MonoCoP is marginal (20.64 to 21.31).
• Training uses synthetic depth perturbations on GT boxes, so distribution shift to real detector residuals is a concern.

Experimental setup

Datasets: Omni3D KITTI nuScenes OVMono3D novel category split

Benchmarks: Refine3D KITTI val Omni3D

Metrics: DirAcc FullAcc DepthErr AP3D

Baselines: MonoDETR MonoCD FD3D MonoMAE MonoTAKD MonoDGP MonoCoP Cube R-CNN OVMono3D DetAny3D LabelAny3D Qwen3-VL-8B

Builds on: Qwen3-VL Omni3D MonoCoP DetAny3D LabelAny3D

Closest papers by meaning

• MonoDGP: Monocular 3D Object Detection with Decoupled-Query and Geometry-Error Priors (CVPR 2025 · similarity 0.89)
• MonoCD: Monocular 3D Object Detection with Complementary Depths (CVPR 2024 · similarity 0.88)
• MonoMAE: Enhancing Monocular 3D Detection through Depth-Aware Masked Autoencoders (NeurIPS 2024 · similarity 0.88)
• LabelAny3D: Label Any Object 3D in the Wild (NeurIPS 2025 · similarity 0.88)
• MonoPlace3D: Learning 3D-Aware Object Placement for 3D Monocular Detection (CVPR 2025 · similarity 0.88)
• A Simple yet Universal Framework for Depth Completion (NeurIPS 2024 · similarity 0.88)