Research map / Graph learning / Knowledge and temporal graphs
Hydra-SGG: Hybrid Relation Assignment for One-stage Scene Graph Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Hydra-SGG combines one-to-one and IoU-based one-to-many relation assignment with an auxiliary self-attention-free decoder (Hydra Branch) to speed up DETR-based one-stage scene graph generation.
Problem
DETR-based one-stage scene graph generation converges slowly because Hungarian matching gives each ground-truth relation only one positive query among many relation queries, and suboptimal but plausible queries are treated as negatives.
Approach
A Hybrid Relation Assignment adds an IoU and class-probability-based one-to-many assignment on top of one-to-one Hungarian matching, and an auxiliary decoder (Hydra Branch) that shares parameters with the main decoder but removes self-attention so queries produce duplicate predictions that suit one-to-many assignment; the branch is discarded at inference.
Key results
- Hydra-SGG reaches 16.0 mR@50 and 10.6 mR@20 on VG150 test in 12 epochs, versus 12.9 mR@50 for the one-to-one baseline trained 50 epochs.
- On Open Images V6 test, Hydra-SGG achieves a score_wtd of 50.0 in 7 epochs, above EGTR's 48.6.
- On GQA test, Hydra-SGG achieves 12.5 mR@50 and 15.6 mR@100 (the paper's text reports 12.7 and 15.9 in the abstract and body, so the table and text disagree).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add Hybrid Relation Assignment (ablation) | Baseline · VG150 test | mR@50: 12.9 -> 14.9 | Helped |
| add Hydra Branch with self-attention removed (ablation) | Vanilla Hydra-SGG · VG150 test | mR@50: 14.9 to 16.0 | Helped |
| set One-to-Many threshold T to 0.4 instead of 0.3, 0.5, or 0.6 (ablation) | Hydra-SGG · VG150 test | mR@50: 14.9; 15.5; 14.9 -> 16.0 | Helped |
| train for 21 epochs instead of 12 (ablation) | Hydra-SGG · VG150 test | mR@50: 16.0 -> 16.1 | Helped |
| train for 24 epochs instead of 12 (ablation) | Hydra-SGG · VG150 test | mR@50: 16.0 -> 15.2 | Hurt |
| use 200 relation queries instead of 300 (ablation) | Hydra-SGG · VG150 test | mR@50: 16.0 -> 15.3 | Hurt |
| use 400 relation queries instead of 300 (ablation) | Hydra-SGG · VG150 test | mR@50: 16.0 -> 15.5 | Hurt |
| reduce One-to-Many loss weight to 0.5 relative to One-to-One (ablation) | Hydra-SGG · VG150 test | mR@50: 16.0 -> 14.6 | Hurt |
| increase One-to-Many loss weight to 3 relative to One-to-One (ablation) | Hydra-SGG · VG150 test | mR@50: 16.0 -> 15.4 | Hurt |
| remove self-attention from the decoder (ablation) | Vanilla Hydra-SGG · VG150 val | average diversity score: 6.6 -> 4.6 | Hurt |
| remove self-attention from query processing (ablation) | Hydra-SGG · VG150 val | mean query distance: 7.55 to 7.23 | Hurt |
Surprising findings
- Removing self-attention between relation queries increases duplicate relation predictions (ADS 4.6 with self-attention off vs 6.6 on), which the paper exploits rather than avoids.
- Vanilla Hybrid Relation Assignment without the Hydra Branch conflicts with self-attention, as one-to-many matching clashes with self-attention's diversity-promoting effect.
Limitations
- Closed-set prediction: cannot recognize object or relation categories unseen during training.
- One-to-many assignment rules are manually designed and may introduce training noise.
- Reported unbiasing comparisons exclude debiasing techniques, so comparisons are against non-debiased methods only.
Experimental setup
Datasets: Visual Genome (VG150) Open Images V6 GQA
Benchmarks: VG150 Open Images V6 GQA
Metrics: mR@20 mR@50 mR@100 R@50 wmAP_rel wmAP_phr score_wtd F-Recall ADS
Baselines: MOTIFS VCTree-TDE BGNN PE-Net IS-GGT VETO UniVRD DRM SGTR SSR-CNN ISG RelTR
Builds on: DETR RelTR Deformable DETR Focal loss Hungarian matching
Closest papers by meaning
- EGTR: Extracting Graph from Transformer for Scene Graph Generation (CVPR 2024 · similarity 0.90)
- Leveraging Predicate and Triplet Learning for Scene Graph Generation (CVPR 2024 · similarity 0.89)
- DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation (CVPR 2024 · similarity 0.88)
- Adaptive Self-training Framework for Fine-grained Scene Graph Generation (ICLR 2024 · similarity 0.88)
- RA-SGG: Retrieval-Augmented Scene Graph Generation Framework via Multi-Prototype Learning (AAAI 2025 · similarity 0.88)
- Scene Graph-Grounded Image Generation (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.