Research map / 3D vision / 3D generation

RelationVGGT : Visual Geometry Transformers for 3D Spatial Relation Segmentation

NeurIPS 2026 · 3D vision › 3D generation · approach: Semantic scene completion · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces 3D spatial relation segmentation and RelationVGGT, a feed-forward, pose-free model that fuses DINOv2 and Pi3 features with a subject-conditioned relation transformer, trained on a VLM/LLM-annotated ScanNet++ relation dataset.

Problem

Localizing, across unposed multi-view images, the target object that satisfies a spatial relation (e.g., "resting on") with respect to a visually specified subject, without per-scene optimization or known camera poses.

Approach

Reformulates volumetric relation search as per-pixel, subject-conditioned relation prediction on target-view grids, using a relation transformer that alternates subject-conditioned cross-attention with multi-view self-attention over fused 2D semantic and 3D geometry features; training uses multi-query focal and Dice losses with an automatically annotated ScanNet++ relation dataset.

Key results

• Average mIoU 0.382 and mAcc 0.549 across Replica, LERF and ScanNet++, vs 0.303 / 0.480 for RelationField.
• On ScanNet++ mIoU 0.445 vs 0.307 for RelationField; on LERF RelationField (0.276) slightly exceeds ours (0.254).
• Human audit: filtering raises proposal precision from 36.4% to 67.2% and removes 92.0% of incorrect proposals.

Surprising findings

• On LERF, the method is comparable to or slightly below RelationField, which has access to ground-truth category names.
• Pointmap positional encoding and DINOv2-only inputs both underperform the fused features; pi3-only is also below the full model.

Limitations

• Data pipeline depends on instance-level annotated datasets such as ScanNet++.
• Retained relations remain noisy (about 67% precision) and coverage is low (28.5% correct retention).
• Automatic subject grounding degrades with subject-instance ambiguity (GroundingDINO+SAM2).

Experimental setup

Datasets: ScanNet++ LERF Replica

Benchmarks: Replica LERF ScanNet++

Metrics: mIoU mAcc@0.25

Baselines: LangSplat OpenGaussian RelationField RelationField† VideoLISA SA2VA UniPixel MVGGT ReferSplat

Builds on: VGGT Pi3 DINOv2 RelationField DPT

Closest papers by meaning

• Can Transformers Capture Spatial Relations between Objects? (ICLR 2024 · similarity 0.89)
• DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation (CVPR 2024 · similarity 0.87)
• AugRefer: Advancing 3D Visual Grounding via Cross-Modal Augmentation and Spatial Relation-based Referring (AAAI 2025 · similarity 0.87)
• Relation3D : Enhancing Relation Modeling for Point Cloud Instance Segmentation (CVPR 2025 · similarity 0.87)
• RelationField: Relate Anything in Radiance Fields (CVPR 2025 · similarity 0.87)
• Towards CLIP-driven Language-free 3D Visual Grounding via 2D-3D Relational Enhancement and Consistency (CVPR 2024 · similarity 0.87)