Research map / Detection and segmentation / Open-set and few-shot detection
Groupwise Query Specialization and Quality-Aware Multi-Assignment for Transformer-based Visual Relationship Detection
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SpeaQ, which trains specialized Transformer queries by grouping queries and predicates and assigns each ground truth to multiple high-quality predictions, improving scene graph generation and HOI detection with no inference overhead.
Problem
Hungarian-matching label assignment for Transformer-based visual relationship detection trains unspecialized queries (each must detect all relations) and suppresses near-correct predictions by assigning them 'no relation' as ground truth.
Approach
Divide queries and predicate classes into disjoint frequency-based groups so each query group only matches relations in its paired predicate group (groupwise query specialization), and assign each GT to multiple predictions chosen adaptively by subject, object, and predicate quality (quality-aware multi-assignment).
Key results
- On VG with ISG, SpeaQ raises R@100 from 32.1 to 36.0 and mR@100 from 8.4 to 14.1 (AvgR@100 20.3 to 25.1).
- On HICO-DET with GEN, SpeaQ improves full mAP from 33.12 to 34.00 and rare mAP by +3.08 (27.12 to 30.20).
- Applied to HOTR on VG, SpeaQ improves R@100 by +2.0 and mR@100 by +3.0 with no added parameters or inference cost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SpeaQ to ISG* | ISG* · Visual Genome | R@100: 30.1 -> 35.5 (+5.4) | Helped |
| Add SpeaQ to ISG* | ISG* · Visual Genome | mR@100: 16.6 -> 17.6 (+1.0) | Helped |
| Add SpeaQ to HOTR | HOTR* · Visual Genome | R@100: 27.1 -> 29.1(+2.0) | Helped |
| Add SpeaQ to HOTR | HOTR* · Visual Genome | mR@100: 9.7 -> 12.7(+3.0) | Helped |
| Add SpeaQ to GEN | GEN* · HICO-DET full | mAP: 33.12 -> 34.00(+0.88) | Helped |
| Add SpeaQ to GEN | GEN* · HICO-DET rare | mAP: 27.12 -> 30.20(+3.08) | Helped |
| Add SpeaQ to GEN | GEN* · HICO-DET non-rare | mAP: 34.91 -> 35.13(+0.22) | Helped |
| Add quality-aware multi-assignment (ablation) | ISG baseline · Visual Genome | R@100: 32.1 -> 33.1 | Helped |
| Add quality-aware multi-assignment (ablation) | ISG baseline · Visual Genome | mR@100: 8.4 -> 9.2 | Helped |
| Add groupwise query specialization (ablation) | ISG baseline · Visual Genome | R@100: 32.1 -> 35.3 | Helped |
| Add groupwise query specialization (ablation) | ISG baseline · Visual Genome | mR@100: 8.4 -> 13.5 | Helped |
| Increase query count from 300 to 600 under conventional training (ablation) | Baseline · Visual Genome | mR@100: 8.4 -> 7.8(-0.6) | Hurt |
Surprising findings
- An IoU-based label assignment commonly used for CNN detectors completely fails when training Transformer-based VRD models (AvgR@100 of 4.1 vs 20.3 for baseline).
- Naively increasing the number of queries under the conventional assignment lowers mR@100 (8.4 to 7.8), whereas SpeaQ allows gains from more queries.
Limitations
- Gains on HICO-DET non-rare set are small (+0.22 mAP).
- Performance declines when the number of predicate groups is too large due to insufficient GTs per group.
Experimental setup
Datasets: Visual Genome (VG150) HICO-DET
Benchmarks: Visual Genome (VG150) HICO-DET
Metrics: R@K mR@K AvgR@K F@K mAP
Baselines: Motifs VCTree RelDN BGNN AS-Net SGTR HOTR ISG GEN SHA
Builds on: DETR ISG HOTR GEN
Closest papers by meaning
- Large-Scale Visual Relationship Understanding (AAAI 2019 · similarity 0.88)
- RelTransformer: A Transformer-Based Long-Tail Visual Relationship Recognition (CVPR 2022 · similarity 0.88)
- Tensorize, Factorize and Regularize: Robust Visual Relationship Learning (CVPR 2018 · similarity 0.88)
- Deep Structured Learning for Visual Relationship Detection (AAAI 2018 · similarity 0.87)
- Visual Relationship Detection with Low Rank Non-Negative Tensor Decomposition (AAAI 2020 · similarity 0.87)
- DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.