Research map / Graph learning / Knowledge and temporal graphs
DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DSGG, a single-stage transformer that predicts scene graphs as dense directed graphs using per-node graph-aware queries, relaxed sub-graph matching, relation distillation and re-scoring, evaluated on Visual Genome and PSG.
Problem
Scene graph generation struggles with incomplete labels, long-tailed relation categories, and relational semantic overlap (multiple predicates between the same object pair); existing transformer methods use separate object/predicate queries or per-triplet queries that limit capacity for dense and ra
Approach
Each graph-aware query encodes a node plus all its outgoing relations; pairwise relation logits are predicted densely with sigmoid MLPs over concatenated query pairs, trained with a relaxed (linearized upper-bound) sub-graph matching solved by the Hungarian algorithm, plus a learned relation filter and object-confidence re-scoring.
Key results
- On Visual Genome SGDet, DSGG with logit adjustment reports mR@50/100 of 20.2/25.5 versus 16.7/18.8 for PE-Net (the strongest prior mean-recall baseline in Table 1 for that setting).
- On PSG (Swin-L) the model reaches mR@50 of 41.7 and mR@100 of 47.8 in panoptic scene graph detection, compared with 37.6 and 40.9 for HiLo.
- On the 927-image PSG relational-overlap subset (R50), DSGG improves R@100 from 51.1 to 63.6 over HiLo.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add relation rescoring (ablation) | DSGG without relation rescoring or distillation · Visual Genome test, SGDet | R@50: 6.9 -> 25.9 | Helped |
| Add relation rescoring (ablation) | DSGG without relation rescoring or distillation · Visual Genome test, SGDet | mR@50: 11.9 -> 12.5 | Helped |
| Add relation distillation (ablation) | DSGG with relation rescoring · Visual Genome test, SGDet | R@50: 25.9 -> 32.9 | Helped |
| Add relation distillation (ablation) | DSGG with relation rescoring · Visual Genome test, SGDet | mR@50: 12.5 -> 13.0 | Helped |
| Add logit adjustment (ablation) | DSGG with relation rescoring and distillation · Visual Genome test, SGDet | mR@50: 13.0 -> 20.2 | Helped |
| Add logit adjustment (ablation) | DSGG with relation rescoring and distillation · Visual Genome test, SGDet | R@50: 32.9 -> 26.5 | Hurt |
| Use graph-aware queries and dense graph prediction | DSGG · PSG test, relational semantic overlap subset | mR@50: +12.4 | Helped |
| Use graph-aware queries and dense graph prediction | DSGG · PSG test, low-frequency relations | mR@50: (+14.0) | Helped |
Surprising findings
- Logit adjustment raises mean recall substantially but slightly lowers plain recall on Visual Genome (e.g., R@50 from 32.9 to 26.5 in ablation).
Limitations
- Relies on Mask2Former initialization and additional relation labels from a baseline model for the best PSG results (dagger variants)
- Sub-graph matching is an approximation (upper bound of an NP-hard quadratic assignment problem)
Experimental setup
Datasets: Visual Genome PSG
Benchmarks: Visual Genome PSG
Metrics: R@50 R@100 mR@50 mR@100 mean@K zero-shot recall SGDet SGCls PredCls
Baselines: Motifs VCTree BGNN SGTR SS-RCNN SHA-GCL NICE HL-Net RU-Net Relationformer RepSGG HetSGG
Builds on: DETR Mask2Former PSGTR HiLo Unbiased SGG (TDE)
Closest papers by meaning
- Scene Graph Generation by Iterative Message Passing (CVPR 2017 · similarity 0.92)
- EGTR: Extracting Graph from Transformer for Scene Graph Generation (CVPR 2024 · similarity 0.91)
- Scene Graph-Grounded Image Generation (AAAI 2025 · similarity 0.90)
- Leveraging Predicate and Triplet Learning for Scene Graph Generation (CVPR 2024 · similarity 0.89)
- Object-Centric Representation Learning for Enhanced 3D Semantic Scene Graph Prediction (NeurIPS 2025 · similarity 0.89)
- From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.