Research map / 3D vision / 3D generation
SIU3R: Simultaneous Scene Understanding and 3D Reconstruction Beyond Feature Alignment
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SIU3R, an alignment-free feed-forward framework that jointly predicts pixel-aligned 3D Gaussians and multi-task segmentation masks from unposed images via unified learnable queries, with two mutual-benefit modules.
Problem
Simultaneous 3D reconstruction and scene understanding from unposed images, where existing methods rely on 2D-to-3D feature alignment that limits instance-level understanding and loses semantic information through feature compression.
Approach
Replace 2D feature alignment with pixel-aligned 2D-to-3D mask lifting: a single set of learnable queries shared across views and tasks decodes cross-view masks that are paired with pixel-aligned 3D Gaussians; a training-free multi-view mask aggregation (reconstruction helps understanding) and a mask-guided depth continuity loss (understanding helps reconstruction) make the two tasks benefit each o
Key results
- Achieves AbsRel 0.07421 and PSNR 25.96 on ScanNet depth and novel view synthesis, versus LSM's 0.07468 and 21.88.
- Reaches 0.5922 mIoU (semantic), 0.2817 mAP and 0.6612 PQ on context-view 2D-only understanding, above LSM and Mask2Former baselines.
- Multi-view mask aggregation is training-free and improves 2D-only mIoU from 0.5512 to 0.5922 while preserving reconstruction quality.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Multi-View Mask Aggregation (R→U) (ablation) | SIU3R · ScanNet context views | mIoU_s: 0.5512 -> 0.5922 | Helped |
| add Multi-View Mask Aggregation (R→U) (ablation) | SIU3R · ScanNet novel views (3D-aware) | mIoU_s: - -> 0.5920 | Helped |
| add Mask-Guided Geometry Refinement (U→R) (ablation) | SIU3R · ScanNet | AbsRel: 0.09619 -> 0.07421 | Helped |
| add Mask-Guided Geometry Refinement (U→R) (ablation) | SIU3R · ScanNet | PSNR: 25.51 -> 25.96 | Helped |
| use early feature aggregation without retraining instead of mask aggregation (ablation) | SIU3R · ScanNet context views | mIoU_s: 0.5922 -> 0.1126 | Hurt |
| use early feature aggregation with retraining instead of mask aggregation (ablation) | SIU3R · ScanNet context views | mIoU_s: 0.5922 -> 0.5442 | Hurt |
| increase input views from 2 to 8 | SIU3R · ScanNet novel-view 3D-aware segmentation | mIoU: 0.592 -> 0.620 | Helped |
| increase input views from 2 to 8 | SIU3R · ScanNet novel-view 3D-aware segmentation | PQ: 0.650 -> 0.697 | Helped |
Surprising findings
- Early feature-level aggregation with training actually lowers 2D and 3D understanding scores (mIoU 0.5442) and raises VRAM to about 33GB, compared with the proposed mask-level aggregation.
Limitations
- Trained on limited data compared with DUSt3R and VGGT, which hurts generalization to broader domains.
- ScanNet cross-view mask ground truth is derived from projecting noisy 3D point clouds, so label quality is poor.
Experimental setup
Datasets: ScanNet COCO
Benchmarks: ScanNet
Metrics: AbsRel RMSE PSNR SSIM LPIPS mIoU mAP PQ
Baselines: pixelSplat MVSplat NoPoSplat Mask2Former LSeg LSM Feature-3DGS NeRF-DFF DUSt3R MASt3R VGGT
Builds on: MASt3R LSM MaskFormer NoPoSplat CLIP
Closest papers by meaning
- Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning (CVPR 2025 · similarity 0.88)
- A Unified Framework for 3D Scene Understanding (NeurIPS 2024 · similarity 0.88)
- VLM-3R: Vision-Language Models Augmented with Instruction-Aligned 3D Reconstruction (CVPR 2026 · similarity 0.87)
- Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering (NeurIPS 2026 · similarity 0.87)
- Large Spatial Model: End-to-end Unposed Images to Semantic 3D (NeurIPS 2024 · similarity 0.87)
- MESC-3D:Mining Effective Semantic Cues for 3D Reconstruction from a Single Image (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.