Research map / Detection and segmentation / 3D detection and driving
Cubify Anything: Scaling Indoor 3D Object Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CA-1M, an exhaustively labeled class-agnostic indoor 3D box dataset rendered per frame from FARO-registered ARKitScenes captures, and CuTR, a fully Transformer 2D-to-3D box detector that outperforms point-based methods on CA-1M and benefits from CA-1M pretraining on SUN RGB-D.
Problem
Indoor 3D object detection from a single RGB(-D) frame is limited by small, entangled, coarsely labeled datasets dominated by room-defining objects, and point/voxel-based detectors rely on noisy depth and 3D inductive biases that may not scale.
Approach
Annotate 3D boxes for nearly all objects directly on high-accuracy FARO laser scans, render them pixel-accurately into every handheld frame, and pair this with a pure Transformer detector (CuTR) that predicts 3D boxes from 2D image features with no 3D inductive biases.
Key results
- CuTR (RGB-D) achieves 62.3% AR25 on CA-1M, about 10 points above the best point-based method (FCAF at 49.5%... with TR3D+FF at 52.9%).
- CuTR RGB-only reaches 35.4% AR25 on CA-1M, versus 22.8% for ImVoxelNet and 20.1% for Cube R-CNN.
- Pretraining CuTR (RGB-D) on CA-1M and fine-tuning on Omni3D SUN RGB-D raises AP25 to 35.8 and AR25 to 73.6, surpassing point-based methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CuTR (RGB-D) instead of the strongest listed point-based baseline | indoor 3D detection models · CA-1M | AR25: 49.5 -> 62.3 | Helped |
| Use CuTR (RGB-D) instead of the strongest listed point-based baseline | indoor 3D detection models · CA-1M | AP25: 29.3 -> 40.9 | Helped |
| Use CuTR (RGB-D) instead of the strongest listed point-based baseline | indoor 3D detection models · Omni3D SUN RGB-D | AP25: 29.1 -> 30.3 | Helped |
| Use CuTR (RGB-D) instead of the strongest listed point-based baseline | indoor 3D detection models · Omni3D SUN RGB-D | AR25: 64.2 -> 60.2 | Hurt |
| Use CuTR (RGB-D) instead of the strongest listed point-based baseline | indoor 3D detection models · SUN RGB-D traditional | AP25: 68.8 -> 59.4 | Hurt |
| Use high-resolution FARO-derived depth rather than onboard LiDAR depth (ablation) | CuTR (RGB-D) on CA-1M · CA-1M | AP25: 40.9 -> 46.7 | Helped |
| Use high-resolution FARO-derived depth rather than onboard LiDAR depth (ablation) | CuTR (RGB-D) on CA-1M · CA-1M | AR25: 62.3 -> 67.2 | Helped |
| Pretrain on CA-1M before fine-tuning (ablation) | CuTR (RGB-D) · Omni3D SUN RGB-D | AR25: +13.4 | Helped |
| Pretrain on CA-1M instead of ARKitScenes (ablation) | CuTR (RGB-D) before fine-tuning on Omni3D SUN RGB-D · Omni3D SUN RGB-D | AP25: 28.3 -> 35.8 | Helped |
| Pretrain on CA-1M instead of ScanNet++ (ablation) | CuTR (RGB-D) before fine-tuning on Omni3D SUN RGB-D · Omni3D SUN RGB-D | AP25: 31.4 -> 35.8 | Helped |
Surprising findings
- Point-based methods narrow their gap to CuTR substantially when trained on high-resolution FARO ground-truth depth, suggesting noisy LiDAR depth hurts point-based models more.
- On the traditional SUN RGB-D split, point-based methods outperform CuTR, and CuTR only surpasses them after CA-1M pretraining.
Limitations
- Predicts only yaw orientation; non-zero pitch and roll are left to future work.
- Ground truth derived from stationary laser scans can miss regions due to occlusion or transparent/reflective objects.
- Evaluation limited to 100 detections per frame and reported at 3D IoU 0.25 and 0.50.
Experimental setup
Datasets: CA-1M ARKitScenes SUN RGB-D Omni3D SUN RGB-D ScanNet++
Benchmarks: CA-1M SUN RGB-D Omni3D SUN RGB-D
Metrics: AP25 AR25 AP50 AR50
Baselines: FCAF TR3D TR3D + FF ImVoxelNet Cube R-CNN
Builds on: Plain DETR MultiMAE Depth-Anything Cube R-CNN ARKitScenes
Closest papers by meaning
- UniDet3D: Multi-dataset Indoor 3D Object Detection (AAAI 2025 · similarity 0.90)
- Uncertainty Meets Diversity: A Comprehensive Active Learning Framework for Indoor 3D Object Detection (CVPR 2025 · similarity 0.89)
- Rooms from Motion: Un-posed Indoor 3D Object Detection as Localization and Mapping (NeurIPS 2025 · similarity 0.89)
- SPGroup3D: Superpoint Grouping Network for Indoor 3D Object Detection (AAAI 2024 · similarity 0.88)
- ImOV3D: Learning Open Vocabulary Point Clouds 3D Object Detection from Only 2D Images (NeurIPS 2024 · similarity 0.88)
- UNION: Unsupervised 3D Object Detection using Object Appearance-based Pseudo-Classes (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.