Research map / Detection and segmentation / CNN features
EGTR: Extracting Graph from Transformer for Scene Graph Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EGTR, a lightweight one-stage scene graph generator that extracts relations from DETR decoder self-attention by-products, trained with adaptive relation smoothing and a connectivity prediction auxiliary task.
Problem
One-stage scene graph generation (SGG) is complex and ignores the inherent object-query relationships already learned in the multi-head self-attention of DETR-style object detectors, while triplet-based models miss objects without relations.
Approach
Treat attention queries and keys from all decoder self-attention layers (plus final hidden states) as subject and object representations, pairwise-concatenate them, gate-sum across layers, and predict predicates with a shallow MLP. Adaptive relation smoothing scales GT relation labels by detection quality, giving a curriculum from detection to multi-task learning, and a connectivity head filters p
Key results
- On Visual Genome, EGTR achieves AP50 of 30.8 (highest object detection) with 42.5M parameters and 14.7 FPS, versus SSR-CNN's 23.8 AP50 at 274.3M params.
- EGTR reports R@50 of 23.5 and mR@50 of 7.9 on Visual Genome; with logit adjustment (tau=0.3) it reaches mR@50 of 14.0.
- On Open Images V6, EGTR attains a score of 48.6, below SSR-CNN's 49.4 but with higher wmAP_rel (42.0 vs 41.5).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add attention query/key representations alongside final hidden states (ablation) | EGTR · Visual Genome | R@50: 29.9 -> 30.2 | Helped |
| Add attention query/key representations alongside final hidden states (ablation) | EGTR · Visual Genome | mR@50: 7.6 -> 7.9 | Helped |
| Use hidden states instead of attention query/key representations across layers (ablation) | EGTR · Visual Genome | R@50: 30.2 -> 29.6 | Hurt |
| Use hidden states instead of attention query/key representations across layers (ablation) | EGTR · Visual Genome | mR@50: 7.9 -> 7.4 | Hurt |
| Add adaptive smoothing (ablation) | EGTR · Visual Genome | R@50: 26.6 -> 28.3 | Helped |
| Add adaptive smoothing (ablation) | EGTR · Visual Genome | mR@50: 5.3 -> 6.5 | Helped |
| Add connectivity prediction (ablation) | EGTR · Visual Genome | R@50: 26.6 -> 29.6 | Helped |
| Add connectivity prediction (ablation) | EGTR · Visual Genome | mR@50: 5.3 -> 7.0 | Helped |
| Add negative and non-matching sampling (ablation) | EGTR · Visual Genome | R@50: 26.6 -> 28.9 | Helped |
| Add negative and non-matching sampling (ablation) | EGTR · Visual Genome | mR@50: 5.3 -> 7.1 | Helped |
| Add hard negative sampling (ablation) | EGTR · Visual Genome | R@50: 29.6 -> 30.2 | Helped |
| Add hard negative sampling (ablation) | EGTR · Visual Genome | mR@50: 7.7 -> 7.9 | Helped |
Surprising findings
- Using only attention by-products yields performance similar to using last-layer hidden states, and simply adding all-layer hidden states decreases triplet detection performance.
Limitations
- On Open Images V6 the overall score is slightly below SSR-CNN (48.6 vs 49.4).
- Triplet detection (R@k) is comparable rather than best; without logit adjustment mR@k is lower than some baselines.
- Relies on a pre-trained object detector trained on the target dataset before SGG training.
Experimental setup
Datasets: Visual Genome Open Images V6
Benchmarks: Visual Genome Open Images V6
Metrics: R@20 R@50 R@100 mR@20 mR@50 mR@100 AP50 FPS micro-R@50 wmAP
Baselines: RelTR SGTR Relationformer Iterative SGG SSR-CNN IMP VCTree GPS-Net BGNN FCSGG Motifs
Builds on: DETR Deformable DETR Motifs SSR-CNN logit adjustment Graph-RCNN
Closest papers by meaning
- DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation (CVPR 2024 · similarity 0.91)
- Hydra-SGG: Hybrid Relation Assignment for One-stage Scene Graph Generation (ICLR 2025 · similarity 0.90)
- Not All Relations Are Equal: Mining Informative Labels for Scene Graph Generation (CVPR 2022 · similarity 0.90)
- IS-GGT: Iterative Scene Graph Generation With Generative Transformers (CVPR 2023 · similarity 0.90)
- Scene Graph-Grounded Image Generation (AAAI 2025 · similarity 0.88)
- Adaptive Visual Scene Understanding: Incremental Scene Graph Generation (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.