Research map / 3D vision / 3D generation
Multiview Scene Graph
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Multiview Scene Graph (MSG) generation from unposed images, a dataset built from ARKitScenes, an IoU-based graph metric, and AoMSG, a Transformer-decoder baseline that jointly learns place and object embeddings on frozen DINOv2 features.
Problem
Building a topological place+object scene graph from unposed RGB images requires joint visual place recognition and object association across limited fields of view and large viewpoint changes; existing VPR and tracking metrics do not evaluate graph quality directly.
Approach
Jointly learn place and object embeddings in a single Transformer decoder over frozen pretrained image tokens (DINOv2 plus a detector), treating the whole image as a place "object", and build the graph by thresholding cosine similarity; evaluate with an IoU over graph adjacency matrices.
Key results
- AoMSG-4 achieves Recall@1 98.3, PP IoU 42.2 and PO IoU 74.2 with GT detections, versus 97.1/35.6 for SALAD on place metrics and 17.4 for UniTrack on PO IoU.
- With GroundingDINO detections, AoMSG-2 and AoMSG-4 both reach PO IoU 28.1, above DEVA (16.6) and UniTrack (13.0).
- In a pilot study on one 22-image scene, GPT-4o obtained PO IoU 85.0 but PP IoU of only 59.3 (63.0 adjusted), noted as a small-scale pilot.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use AoMSG-4 instead of SALAD for place recognition | ARKitScenes test · ARKitScenes test | PP IoU: 35.6 -> 42.2 | Helped |
| Use AoMSG-4 instead of SALAD for place recognition | ARKitScenes test · ARKitScenes test | Recall@1: 97.1 -> 98.3 | Helped |
| Use AoMSG-4 instead of UniTrack for object association with ground-truth detections | ARKitScenes test · ARKitScenes test | PO IoU: 17.4 -> 74.2 | Helped |
| Use AoMSG-4 instead of DEVA with GroundingDINO detections | ARKitScenes test · ARKitScenes test | PO IoU: 16.6 -> 28.1 | Helped |
| Replace ground-truth detections with GroundingDINO detections (ablation) | AoMSG-4 · ARKitScenes test | PO IoU: 74.2 -> 28.1 | Hurt |
| Increase Transformer decoder depth from 2 to 4 layers (ablation) | AoMSG · ARKitScenes test | PP IoU: 40.7 -> 42.2 | Helped |
| Increase Transformer decoder depth from 2 to 4 layers (ablation) | AoMSG · ARKitScenes test | PO IoU (GT detection): 69.1 -> 74.2 | Helped |
| Vary projector dimension among 512, 1024, and 2048 (ablation) | AoMSG-4 · ARKitScenes test | PP IoU: 41.3; 42.2; 41.8 | No clear effect |
| Vary projector dimension among 512, 1024, and 2048 (ablation) | AoMSG-4 · ARKitScenes test | PO IoU: 72.9; 74.2; 72.4 | No clear effect |
Surprising findings
- High place Recall@1 (above 96% for all methods) does not guarantee good graph quality, with PP IoU below 50 for all methods.
- Performance saturates with DINOv2 model size, which the authors attribute possibly to limited data.
Limitations
- Experiments use only one dataset (ARKitScenes) with static objects.
- Object detection quality is not studied; detector choice affects PO IoU considerably.
- No downstream evaluation such as navigation is performed.
Experimental setup
Datasets: ARKitScenes
Benchmarks: MSG dataset (ARKitScenes-derived)
Metrics: Recall@1 PP IoU PO IoU GIoU-based object matching
Baselines: AnyLoc NetVLAD Mickey SALAD UniTrack DEVA SepMSG-Direct SepMSG-Linear SepMSG-MLP
Builds on: DETR SuperGlue DINOv2 supervised contrastive learning NetVLAD
Closest papers by meaning
- Object-Centric Representation Learning for Enhanced 3D Semantic Scene Graph Prediction (NeurIPS 2025 · similarity 0.87)
- EGTR: Extracting Graph from Transformer for Scene Graph Generation (CVPR 2024 · similarity 0.86)
- HIG: Hierarchical Interlacement Graph Approach to Scene Graph Generation in Video Understanding (CVPR 2024 · similarity 0.86)
- DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation (CVPR 2024 · similarity 0.86)
- Scene Graph-Grounded Image Generation (AAAI 2025 · similarity 0.86)
- Uncovering and Shaping the Latent Representation of 3D Scene Topology in Vision-Language Models (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.