Research map / 3D vision / 3D generation
LabelAny3D: Label Any Object 3D in the Wild
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LabelAny3D, an analysis-by-synthesis pipeline that reconstructs 3D scenes from in-the-wild images to auto-label 3D boxes, and releases the COCO3D open-vocabulary monocular 3D detection benchmark built on MS-COCO.
Problem
Monocular 3D object detection on in-the-wild images is limited by the lack of 3D datasets and costly 3D box annotation, especially for categories outside indoor and driving domains.
Approach
Reconstructs holistic 3D scenes from a single image by combining super-resolution, amodal completion, image-to-3D mesh generation (TRELLIS), MoGe relative depth aligned to Depth Pro metric depth, and 2D-3D PnP pose estimation with depth-based scale recovery, then fits 3D boxes from the posed meshes.
Key results
- On COCO3D, OVMono3D fine-tuned on Omni3D plus LabelAny3D labels reaches 10.92 AP3D vs 5.87 for the Omni3D baseline, versus 6.82 for OVM3D-Det labels.
- LabelAny3D pseudo-labels reach 64.17 AP3D against human-refined COCO3D boxes, vs 10.03 for OVM3D-Det.
- Ablation: full pipeline 43.17 AP3D vs 1.95 for vanilla Gen3DSR; removing super-resolution drops to 28.13.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train OVMono3D using Omni3D plus LabelAny3D pseudo-labels | OVMono3D · COCO3D | AP3D: 5.87 -> 10.92 | Helped |
| Train OVMono3D using Omni3D plus LabelAny3D pseudo-labels | OVMono3D · Omni3D Novel | AP3D: 16.05 -> 16.98 | Helped |
| Train OVMono3D using Omni3D plus LabelAny3D pseudo-labels | OVMono3D · Omni3D Base | AP3D: 24.77 -> 22.74 | Hurt |
| Use LabelAny3D rather than OVM3D-Det to produce pseudo annotations | 3D pseudo-annotation pipeline · COCO3D | AP3D: 10.03 -> 64.17 | Helped |
| Use LabelAny3D rather than OVM3D-Det to produce pseudo annotations | 3D pseudo-annotation pipeline · KITTI | AP3D: 12.39 -> 13.6 | Helped |
| Use LabelAny3D for highly visible objects rather than OVM3D-Det | 3D pseudo-annotation pipeline · SUN-RGBD, nuScenes, and Objectron | AP3D: 37.58 -> 37.71; 11.38 -> 13.59; 3.84 -> 6.07 | Helped |
| Use LabelAny3D for low-visibility objects rather than OVM3D-Det | 3D pseudo-annotation pipeline · SUN-RGBD, nuScenes, and Objectron | AP3D: 25.56 -> 20.20; 8.50 -> 6.69; 3.85 -> 2.54 | Hurt |
| Use OVM3D-Det for low-visibility objects and LabelAny3D for high-visibility objects | Ensembled annotation pipeline · SUN-RGBD, nuScenes, and Objectron | AP3D: 29.89 -> 30.23; 8.22 -> 9.12; 3.73 -> 4.74 | Helped |
| Add image super-resolution (ablation) | LabelAny3D · COCO3D annotation ablation subset | AP3D: 28.13 -> 43.17 | Helped |
| Add amodal completion (ablation) | LabelAny3D · COCO3D annotation ablation subset | AP3D: 39.22 -> 43.17 | Helped |
| Use MoGe relative depth scaled to Depth Pro rather than Depth Pro alone (ablation) | LabelAny3D · COCO3D annotation ablation subset | AP3D: 22.77 -> 43.17 | Helped |
| Use TRELLIS rather than DreamGaussian for 3D reconstruction (ablation) | LabelAny3D · COCO3D annotation ablation subset | AP3D: +6.33 points | Helped |
Surprising findings
- Training OVMono3D from scratch on OVM3D-Det pseudo-labels fails to converge, and fine-tuning on them yields only marginal gains.
- Fine-tuned models degrade on OVMono3D's base categories, attributed to catastrophic forgetting and label noise.
Limitations
- Pseudo-labels inherit errors from foundation models under heavy occlusion, textureless regions, and small objects.
- TRELLIS may produce ambiguous depth along the viewing direction, causing box misalignment.
- Benchmark is not exhaustively annotated; excludes occluded and truncated objects.
Experimental setup
Datasets: MS-COCO COCONut Omni3D
Benchmarks: COCO3D Omni3D SUN RGB-D nuScenes Objectron KITTI
Metrics: AP3D AR3D Relative Layout AP3D IoU3D
Baselines: OVM3D-Det OVMono3D Gen3DSR DreamGaussian
Builds on: OVMono3D Gen3DSR TRELLIS MoGe Cube R-CNN
Closest papers by meaning
- ImOV3D: Learning Open Vocabulary Point Clouds 3D Object Detection from Only 2D Images (NeurIPS 2024 · similarity 0.89)
- Training an Open-Vocabulary Monocular 3D Detection Model without 3D Data (NeurIPS 2024 · similarity 0.89)
- MonoPlace3D: Learning 3D-Aware Object Placement for 3D Monocular Detection (CVPR 2025 · similarity 0.89)
- ImageNet3D: Towards General-Purpose Object-Level 3D Understanding (NeurIPS 2024 · similarity 0.89)
- OpenBox: Annotate Any Bounding Boxes in 3D (NeurIPS 2025 · similarity 0.88)
- RefineAny3D: Depth Refinement as Semantic Alignment for Monocular 3D Detection (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.