Research map / 3D vision / 3D generation
CN-RMA: Combined Network with Ray Marching Aggregation for 3D Indoor Object Detection from Multi-view Images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CN-RMA, an end-to-end combination of Atlas-style TSDF reconstruction and FCAF3D detection, using ray-marching occlusion-aware feature aggregation to lift multi-view image features into 3D for indoor object detection.
Problem
3D indoor object detection from multi-view images is ill-posed because image features cannot be correctly associated with 3D locations without occlusion information; two-stage reconstruct-then-detect pipelines lack end-to-end connectivity.
Approach
Use a rough TSDF from a multi-view reconstruction network to compute NeuS/NeRF-style opacity and transmittance weights along each camera ray, so image features are averaged into 3D points only near the reconstructed surface, then detect objects with a sparse 3D CNN (FCAF3D) trained with a pretrain-then-joint-finetune schedule.
Key results
- On ScanNet, CN-RMA reaches 58.6 mAP@0.25 and 36.8 mAP@0.5, versus 55.4 and 33.8 for the Atlas+FCAF3D two-stage baseline and 54.8 and 28.4 for ImGeoNet.
- On ARKitScenes, CN-RMA reaches 67.6 mAP@0.25 and 56.5 mAP@0.5, versus 60.2 and 43.4 for ImGeoNet.
- Replacing RMA with volume aggregation (VA) drops ScanNet mAP@0.25 from 58.6 to 31.1 and mAP@0.5 from 36.8 to 11.3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CN-RMA instead of ImGeoNet | 3D indoor object detection from multi-view images · ScanNet test | mAP@0.25: 54.8 -> 58.6 | Helped |
| use CN-RMA instead of ImGeoNet | 3D indoor object detection from multi-view images · ScanNet test | mAP@0.5: 28.4 -> 36.8 | Helped |
| use CN-RMA instead of ImGeoNet | 3D indoor object detection from multi-view images · ARKitScenes test | mAP@0.25: 60.2 -> 67.6 | Helped |
| use CN-RMA instead of ImGeoNet | 3D indoor object detection from multi-view images · ARKitScenes test | mAP@0.5: 43.4 -> 56.5 | Helped |
| replace Volume Aggregation with RMA (ablation) | CN-RMA · ScanNet | mAP@0.25: 31.1 -> 58.6 | Helped |
| replace Volume Aggregation with RMA (ablation) | CN-RMA · ScanNet | mAP@0.5: 11.3 -> 36.8 | Helped |
| replace RMA with Depth Aggregation (ablation) | CN-RMA · ScanNet | mAP@0.25: +0.7 | Helped |
| replace RMA with Depth Aggregation (ablation) | CN-RMA · ScanNet | mAP@0.5: +2.3 | Helped |
| use RMA with θ_rma=0.10 instead of θ_rma=0.05 (ablation) | CN-RMA · ScanNet | mAP@0.25: 58.6 -> 57.3 | Hurt |
| use RMA with θ_rma=0.10 instead of θ_rma=0.05 (ablation) | CN-RMA · ScanNet | mAP@0.5: 36.8 -> 35.4 | Hurt |
| add MVS pretraining, detection pretraining, and joint fine-tuning (ablation) | CN-RMA · ScanNet | mAP@0.25: 48.2 -> 58.6 | Helped |
| add MVS pretraining, detection pretraining, and joint fine-tuning (ablation) | CN-RMA · ScanNet | mAP@0.5: 28.8 -> 36.8 | Helped |
Surprising findings
- Naive volume aggregation without occlusion reasoning performs far worse (31.1 mAP@0.25) than the ImVoxelNet-style baseline, showing that unoccluded feature voting is harmful.
- Joint training from scratch reaches only 48.2 mAP@0.25 on ScanNet, so the pretraining stages are necessary.
Limitations
- Baseline numbers for ScanNet and ARKitScenes are cited from ImGeoNet rather than re-run, and the code was not public at the time of the baseline runs.
- RMA samples 300 points per pixel, and smaller weight thresholds are infeasible due to excessive GPU memory use.
- Performance depends on reconstruction quality; ARKitScenes has lower-resolution depth and rougher annotations.
Experimental setup
Datasets: ScanNet ARKitScenes
Benchmarks: ScanNet ARKitScenes
Metrics: mAP@0.25 mAP@0.5
Baselines: ImVoxelNet NeRF-Det ImGeoNet Atlas+FCAF3D NeuralRecon+FCAF3D
Builds on: Atlas FCAF3D NeuS NeRF ImVoxelNet
Closest papers by meaning
- Cubify Anything: Scaling Indoor 3D Object Detection (CVPR 2025 · similarity 0.86)
- Rooms from Motion: Un-posed Indoor 3D Object Detection as Localization and Mapping (NeurIPS 2025 · similarity 0.86)
- UniDet3D: Multi-dataset Indoor 3D Object Detection (AAAI 2025 · similarity 0.85)
- LASA: Instance Reconstruction from Real Scans using A Large-scale Aligned Shape Annotation Dataset (CVPR 2024 · similarity 0.85)
- Towards Generalizable Multi-Camera 3D Object Detection via Perspective Rendering (AAAI 2025 · similarity 0.85)
- MVSDet: Multi-View Indoor 3D Object Detection via Efficient Plane Sweeps (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.