Research map / 3D vision / 3D generation

Geometry-Guided Semantic Reconstruction for 3D Instance Segmentation

NeurIPS 2026 · 3D vision › 3D generation · approach: Semantic scene completion

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes masked semantic reconstruction of background lifted 2D priors plus geometry-modulated (FiLM-style) semantic injection to distill DINO-X 2D features into sparse 3D instance segmentation encoders.

Problem

Injecting rich 2D foundation-model semantics into 3D instance segmentation encoders by simple concatenation is passive and misaligned with 3D geometry, providing no explicit supervision for semantic transfer.

Approach

Mask a subset of background superpoint 2D priors and train the sparse encoder to reconstruct them, while using raw 3D geometry to produce affine (FiLM-like) parameters that modulate the lifted 2D stream before sparse encoding; reconstruction branch is training-only.

Key results

• On ScanNetV2, Relation3D(Ours) reaches 65.0 mAP, 82.8 AP@50, 88.2 AP@25, up from 62.5 mAP for Relation3D.
• On ScanNet200, Relation3D(Ours) reaches 36.6 mAP vs 31.6 for Relation3D, and SegDINO3D(Ours) reaches 35.2 mAP vs 32.6 for SegDINO3D*.
• In 5% and 10% ScanNet200 training data, mAP gains over the baseline are 1.99 and 3.09 for Relation3D and 6.27 and 9.23 for SegDINO3D.

Surprising findings

• Reconstructing all lifted 2D priors without masking gives only marginal gains (64.4 vs 63.7 mAP), while foreground masking drops mAP by 5.9.

Limitations

• Depends on the quality and availability of lifted multi-view 2D priors.
• Evaluation limited to indoor RGB-D benchmarks.
• No error bars from multiple runs reported.

Experimental setup

Datasets: ScanNetV2 ScanNet200

Benchmarks: ScanNetV2 ScanNet200

Metrics: mAP AP@50 AP@25

Baselines: Relation3D SegDINO3D IKNE SphMask EASE Maft QueryFormer SoftGroup ISBNet Mask3D SPFormer

Builds on: Relation3D SegDINO3D FiLM Sparse 3D U-Net DETR

Closest papers by meaning

• Edge-Aware 3D Instance Segmentation Network with Intelligent Semantic Prior (CVPR 2024 · similarity 0.89)
• Segment Any 3D Object with Language (ICLR 2025 · similarity 0.88)
• Merge3D: Efficient 3D Multimodal LLMs via Joint 2D-3D Token Merging (CVPR 2026 · similarity 0.87)
• SAI3D: Segment Any Instance in 3D Scenes (CVPR 2024 · similarity 0.87)
• MESC-3D:Mining Effective Semantic Cues for 3D Reconstruction from a Single Image (CVPR 2025 · similarity 0.87)
• Know Your Neighbors: Improving Single-View Reconstruction via Spatial Vision-Language Reasoning (CVPR 2024 · similarity 0.87)