Research map / Vision-language and multimodal / Cross-modal learning
Universal Scene Graph Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Universal Scene Graph (USG), a unified representation spanning any combination of text, image, video, and 3D inputs, and USG-Par, an end-to-end parser that aligns cross-modal objects and mitigates domain imbalance via text-centric contrastive learning.
Problem
Scene graph research is confined to single modalities (text, image, video, 3D), and no unified representation or parser exists for combining modalities, which faces cross-modal object alignment and domain imbalance issues.
Approach
Define a Universal SG that merges modality-specific and modality-shared objects/relations from any modality combination, and parse it end-to-end with an object associator that projects objects into each other's feature spaces plus a text-centric contrastive loss that aligns other modalities to textual SGs.
Key results
- On PSG SGDet with joint training, USG-Par reaches 36.9/32.1 R/mR@20 and 52.4/44.6 R/mR@100, outperforming Pair-Net (Swin-B) at 42.4/29.7 R/mR@100.
- On 3DDSG SGCls, USG-Par achieves 46.9 R@100 versus 45.7 for CCL-3DDSG.
- On FACTUAL text SG, USG-Par achieves Set Match 43.83 and SPICE 84.38 versus 42.35 and 82.43 for FACTUAL-T5 (pre).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| jointly train USG-Par across single-modality datasets rather than train only on the corresponding dataset | USG-Par · PSG | R@K average: +3.2 | Helped |
| add linear transformation before cross-modal object association (ablation) | USG-Par object associator · Object association, S-I/V/D | Association accuracy@5: 4.6 / 3.8 / 1.7 -> 13.6 / 13.9 / 12.0 | Helped |
| add the filter to the object associator (ablation) | USG-Par object associator · Object association, S-I/V/D | Association accuracy@5: 12.5 / 11.7/ 11.1 -> 13.6 / 13.9 / 12.0 | Helped |
| add supervised association-matrix loss (ablation) | USG-Par object associator · Object association, S-I/V/D | Association accuracy@5: 10.7 / 10.8 / 11.4 -> 13.6 / 13.9 / 12.0 | Helped |
| add subject/object projector to RPC (ablation) | USG-Par RPC · PSG | R/mR@50: 36.5 / 28.5 -> 44.6 / 40.9 | Helped |
| add relation-aware cross-attention to RPC (ablation) | USG-Par RPC · PSG | R/mR@50: 32.5 / 26.5 -> 44.6 / 40.9 | Helped |
| add pair loss to RPC (ablation) | USG-Par RPC · PSG | R/mR@50: 38.4 / 31.4 -> 44.6 / 40.9 | Helped |
| replace MLP relation decoder with a transformer-based decoder (ablation) | USG-Par relation decoder · PSG | R/mR@50: 34.1 / 20.7 -> 44.6 / 40.9 | Helped |
| add cross-attention layers to the relation decoder (ablation) | USG-Par relation decoder · PSG | R/mR@50: 39.6 / 28.4 -> 44.6 / 40.9 | Helped |
| add text-centric scene contrastive learning (ablation) | USG-Par · S-I multimodal | Set Match: 79.6 -> 80.3 | Helped |
| add text-centric scene contrastive learning (ablation) | USG-Par · S-V multimodal | Set Match: 79.4 -> 80.6 | Helped |
| add text-centric scene contrastive learning (ablation) | USG-Par · S-D multimodal | Set Match: 71.1 -> 79.2 | Helped |
Surprising findings
- Removing the projection and relation-aware cross-attention in RPC causes model divergence with no correct predictions, indicating object queries alone lack pairwise information.
Limitations
- Multimodal USG training data is partly synthesized by GPT-4o parsing and label matching, which may introduce noise.
- PVSG and some multimodal results have low absolute scores (e.g., mR@50 of 2.0-2.4 on S-V).
Experimental setup
Datasets: VG PSG AG PVSG 3DDSG FACTUAL
Benchmarks: PSG PVSG 3DDSG FACTUAL VG
Metrics: Recall@K mean Recall@K Set Match SPICE
Baselines: PSGTR HiLo DSGG Pair-Net VL-SAT CCL-3DDSG FACTUAL-T5 SGFN
Builds on: Mask2Former Pair-Net Point-BERT OpenCLIP
Closest papers by meaning
- The Scene Language: Representing Scenes with Programs, Words, and Embeddings (CVPR 2025 · similarity 0.89)
- Scene Graph Generation with Role-Playing Large Language Models (NeurIPS 2024 · similarity 0.89)
- From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models (CVPR 2024 · similarity 0.88)
- Scene Graph-Grounded Image Generation (AAAI 2025 · similarity 0.88)
- Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-Modal Structured Representations (AAAI 2024 · similarity 0.88)
- CrossOver: 3D Scene Cross-Modal Alignment (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.