Research map / Detection and segmentation / CNN features
Scene-Graph ViT: End-to-End Open-Vocabulary Visual Relationship Detection
Tim Salzmann, Markus Ryll, Alex Bewley, Matthias Minderer
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace decoder-based relationship module (UniVRD) with encoder-only Relationship Attention, same data mixture (ablation) | SG-ViT B/32 · VG150 | mR@100: 12.1 -> 17.4 | Improved |
| add simple predicate rebalancing fine-tuning (ablation) | SG-ViT L/14 · VG150 | mR@100: 21.8 -> 26.1 | Improved |
| remove Objects365 detection data from training mixture (ablation) | SG-ViT B/32 · VG150 | mR@100: 18.1 -> 15.3 | Hurt |
| remove Visual Genome from training mixture (ablation) | SG-ViT B/32 · VG150 | mR@100: 18.1 -> 14.2 | Hurt |
| scale backbone from ViT-B/32 to ViT-L/14 (ablation) | SG-ViT · VG150 | mR@100: 18.1 -> 21.8 | Improved |
| reduce top-k relationships at inference from 2^14 to 2^11 (ablation) | SG-ViT · VRD speed and accuracy | accuracy and speed: speed 71-81% -> 90% of object detector | No clear effect |
| share one MLP for subject and object embeddings (ablation) | Relationship Attention · VRD training | final score | Hurt |
| use SG-ViT instead of UniVRD on HICO | HICO-DET · HICO test (L/14) | mAP: 37.41 -> 38.11 | No clear effect |
| use SG-ViT instead of UniVRD on rare HOI classes | HICO-DET · HICO test (L/14) | mAPr: 28.90 -> 33.71 | Improved |
| add relationship attention to the object detector on pure detection data | OWL-ViT · LVIS | mAP (%): 22.1 -> 21.9 | No clear effect |
Closest papers by meaning
- Large-Scale Visual Relationship Understanding (AAAI 2019 · similarity 0.92)
- Visual Relationship Detection With Deep Structural Ranking (AAAI 2018 · similarity 0.92)
- Unified Visual Relationship Detection with Vision and Language Models (ICCV 2023 · similarity 0.91)
- Tensorize, Factorize and Regularize: Robust Visual Relationship Learning (CVPR 2018 · similarity 0.91)
- EGTR: Extracting Graph from Transformer for Scene Graph Generation (CVPR 2024 · similarity 0.90)
- Multi-Modal Prompting for Open-Vocabulary Video Visual Relationship Detection (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
