Research map / 3D vision / 3D generation
EA3D: Online Open-World 3D Object Extraction from Streaming Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EA3D, a unified online framework that jointly reconstructs 3D Gaussians and extracts open-world object semantics (CLIP/Grounded-SAM features, VLM-derived categories) from streaming monocular video without poses or geometric priors.
Problem
Open-world 3D scene understanding typically requires offline multi-view images, known or COLMAP poses, and pre-built 3D geometry; this work targets joint online reconstruction and holistic object extraction from streaming video.
Approach
Embeds knowledge-integrated features from VLMs and 2D foundation encoders (CLIP, Grounded-SAM) into feature Gaussians updated online via feed-forward updates, with online visual odometry (Cut3R-based keypoint graph with bundle adjustment) and recurrent joint semantic-geometry optimization.
Key results
- On ScanNet, EA3D (pose-free) reaches 25.8 PSNR, 46.3 mIoU segmentation and 39.9 bbox mAP, outperforming HiCoM+VFM (22.6 PSNR, 34.8 mIoU) and the offline InstanceGaussian/FeatureGS baselines on reconstruction and segmentation.
- On LERF with 10 sparse views, EA3D achieves 21.9 PSNR versus 18.1 for HiCoM+VFM and 15.2 for FeatureGS, while sustaining about 0.235 FPS online.
- Ablations show removing the online optimization drops training speed from 0.23 to 0.07 FPS and PSNR from 25.8 to 24.6.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add CLIP encoder (ablation) | EA3D · ScanNet | mIoU: 41.6 -> 46.3 | Helped |
| add SAM encoder (ablation) | EA3D · ScanNet | mIoU: 42.8 -> 46.3 | Helped |
| add semantic-awareness regularization (ablation) | EA3D · ScanNet | mIoU: 44.3 -> 46.3 | Helped |
| add online updating (ablation) | EA3D · ScanNet | mIoU: 44.5 -> 46.3 | Helped |
| add online visual odometry (ablation) | EA3D · ScanNet | mIoU: 45.4 -> 46.3 | Helped |
| add joint optimization (ablation) | EA3D · ScanNet | mIoU: 45.7 -> 46.3 | Helped |
| use online self-estimated poses instead of COLMAP-estimated poses | EA3D · ScanNet | PSNR: 25.5 -> 25.8 | Helped |
| use online self-estimated poses instead of COLMAP-estimated poses | EA3D · ScanNet | AP (3D bbox): 59.2 -> 57.9 | Hurt |
| apply EA3D instead of HiCOM+VFM | online reconstruction and segmentation on LERF · LERF, 10 views | mIoU: 36.1 -> 53.8 | Helped |
Surprising findings
- Naive combinations of existing online reconstruction and VFM semantics (e.g., HiCoM+VFM, EmbodiedOcc+L_RGB) perform poorly and can degrade overall performance, per the authors.
Limitations
- Baseline official code was unstable, so baselines were reimplemented by the authors, which may affect comparability.
- No error bars reported due to computational cost.
Experimental setup
Datasets: ScanNet LERF
Benchmarks: ScanNet LERF
Metrics: PSNR SSIM mIoU mAcc AP mAP IoU FPS
Baselines: LangSplat GaussianGrouping FeatureGS OpenGaussian InstanceGaussian OpenScene EmbodiedSAM SAM3D HiCoM+VFM MonoGS+VFM EmbodiedOcc
Builds on: HiCoM Cut3R 3D Gaussian Splatting LangSplat PGSR
Closest papers by meaning
- ImageNet3D: Towards General-Purpose Object-Level 3D Understanding (NeurIPS 2024 · similarity 0.88)
- IGGT4D: Streaming 4D Instance-Grounded Geometry Transformer (NeurIPS 2026 · similarity 0.88)
- Open3DIS: Open-Vocabulary 3D Instance Segmentation with 2D Mask Guidance (CVPR 2024 · similarity 0.87)
- Cross-Modal and Uncertainty-Aware Agglomeration for Open-Vocabulary 3D Scene Understanding (CVPR 2025 · similarity 0.87)
- Uni3D: Exploring Unified 3D Representation at Scale (ICLR 2024 · similarity 0.87)
- Motion4D: Learning 3D-Consistent Motion and Semantics for 4D Scene Understanding (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.