Research map / Point clouds / Point cloud classification
BIP3D: Bridging 2D Images and 3D Perception for Embodied Intelligence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BIP3D, an image-centric 3D perception model built on GroundingDINO that fuses multi-view RGB(-D) images and text with explicit 3D camera position encoding for 3D detection and visual grounding on EmbodiedScan.
Problem
Point-cloud-centric 3D perception suffers from sparsity, noise, and scarce annotated data, limiting performance for embodied intelligence; the paper asks how to exploit 2D vision foundation models for 3D detection and visual grounding.
Approach
Initialize a GroundingDINO-based network with 2D pretrained weights and extend it to multi-view 3D: a spatial enhancer that encodes camera geometry and depth distributions into image features, a 3D deformable aggregation decoder for multi-view fusion, camera intrinsic standardization, and a Wasserstein-distance box loss.
Key results
- On EmbodiedScan validation 3D detection, BIP3D reaches 20.91 overall AP3D@0.25 versus 15.22 for EmbodiedScan-D (+5.69), with RGB-only 17.40.
- On 3D visual grounding validation, BIP3D reaches 54.66 overall AP vs 39.41 for EmbodiedScan (+15.25).
- On the test set, BIP3D with ensemble reaches 62.08 AP, 2.49 above DenseG (59.59); without ensemble it reaches 57.05.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GroundingDINO rather than ImageNet pretraining (ablation) | BIP3D with RGB-D input · EmbodiedScan validation, 3D detection | AP3D@0.25: 14.92 -> 20.91 | Helped |
| add depth maps to RGB input | BIP3D · EmbodiedScan validation, 3D detection | AP3D@0.25: 17.40 -> 20.91 | Helped |
| add camera intrinsic standardization (ablation) | BIP3D with RGB input · EmbodiedScan validation, 3D detection | AP3D@0.25: 16.67 -> 17.40 | Helped |
| add camera intrinsic standardization (ablation) | BIP3D with RGB-D input · EmbodiedScan validation, 3D detection | AP3D@0.25: 19.78 -> 20.91 | Helped |
| replace corner Chamfer distance loss with Wasserstein distance loss (ablation) | BIP3D · EmbodiedScan validation, 3D detection | AP3D@0.25: 0.64 -> 20.91 | Helped |
| replace L1 loss with Wasserstein distance loss (ablation) | BIP3D · EmbodiedScan validation, 3D detection | AP3D@0.25: 17.79 -> 20.91 | Helped |
| replace Wasserstein distance loss with permutation corner distance loss (ablation) | BIP3D · EmbodiedScan validation, 3D detection | AP3D@0.25: 20.21 vs 20.91 | No clear effect |
| randomly sample 1 to 10 descriptions during training rather than use one description (ablation) | BIP3D · EmbodiedScan validation, 3D visual grounding | AP3D@0.25: 53.10 -> 54.66 | Helped |
| randomly sample 1 to 10 descriptions during training rather than use 10 descriptions (ablation) | BIP3D · EmbodiedScan validation, 3D visual grounding | AP3D@0.25: 49.05 -> 54.66 | Helped |
Surprising findings
- Initializing EmbodiedScan with GroundingDINO weights gives only a marginal 1.11% gain, whereas the same initialization gives 5.66-5.99% gains for BIP3D.
- The corner chamfer distance box loss was unstable and hard to converge on BIP3D, while L1 loss is limited to 17.79 AP due to orientation ambiguity.
Limitations
- Evaluated only on EmbodiedScan indoor data; generalization to other benchmarks or outdoor scenes is not shown.
- Performance drops when depth is removed (RGB-only is 3.51 AP lower), and small-object gains depend on depth input.
- Training with more text descriptions per sample introduces a train-test domain gap, requiring random sampling tuned by ablation.
Experimental setup
Datasets: EmbodiedScan
Benchmarks: EmbodiedScan
Metrics: AP3D@0.25
Baselines: VoteNet ImVoxelNet FCAF3D EmbodiedScan SAG3D DenseG GroundingDINO
Builds on: GroundingDINO DINO 3D deformable aggregation PETR EmbodiedScan
Closest papers by meaning
- EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI (CVPR 2024 · similarity 0.90)
- Abstract 3D Perception for Spatial Intelligence in Vision-Language Models (CVPR 2026 · similarity 0.88)
- SPA: 3D Spatial-Awareness Enables Effective Embodied Representation (ICLR 2025 · similarity 0.87)
- Surprise3D: A Dataset for Spatial Understanding and Reasoning in Complex 3D Scenes (NeurIPS 2025 · similarity 0.87)
- D3D-VLP: Dynamic 3D Vision-Language-Planning Model for Embodied Grounding and Navigation (CVPR 2026 · similarity 0.86)
- An Embodied Generalist Agent in 3D World (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.