Research map / Vision-language and multimodal / Human and embodied modeling
Hybrid Reciprocal Transformer with Triplet Feature Alignment for Scene Graph Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a one-stage hybrid reciprocal transformer that jointly decodes triplet mask features and subject/object/predicate components, with a triplet alignment loss and bidirectional refinement, achieving SOTA scene graph generation on Visual Genome and Action Genome.
Problem
Scene graph generation struggles to distinguish overlapping visual relation tuples involving multi-role objects, and predicate recognition suffers from noise because triplet and component features are misaligned or optimized separately.
Approach
Uses triplet mask queries as proposals to guide component-level decoders, adds a triplet-component alignment loss that matches a predicted triplet similarity matrix to the ground-truth matching matrix, and a cross-conditioned bidirectional refinement between triplet and component queries across layers, all trained end-to-end in one stage.
Key results
- On Visual Genome, the method reaches R@50 34.1 and R@100 38.3 with ResNet-101, +1.2 and +2.3 over the prior best one-stage SpeaQISG.
- On Visual Genome, mR@100 rises to 20.5 (+2.9 over SpeaQISG), and F@100 to 26.7 (+3.2).
- On Action Genome, R@20/50 reaches 32.0/40.2 and mR@20/50 21.3/23.6, the best reported in the table.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the hybrid reciprocal transformer with triplet feature alignment | Scene graph generation · Visual Genome | R@50: 32.9 -> 34.1 | Helped |
| use the hybrid reciprocal transformer with triplet feature alignment | Scene graph generation · Visual Genome | R@100: 36.0 -> 38.3 | Helped |
| use the hybrid reciprocal transformer with triplet feature alignment | Scene graph generation · Visual Genome | mR@50: 15.1 -> 16.0 | Helped |
| use the hybrid reciprocal transformer with triplet feature alignment | Scene graph generation · Visual Genome | mR@100: 17.6 -> 20.5 | Helped |
| use the hybrid reciprocal transformer with triplet feature alignment | Scene graph generation · Action Genome | R@20/50: 31.8 / 39.4 -> 32.0 / 40.2 | Helped |
| use the hybrid reciprocal transformer with triplet feature alignment | Scene graph generation · Action Genome | mR@20/50: 20.9 / 23.1 -> 21.3 / 23.6 | Helped |
| use the hybrid reciprocal transformer with triplet feature alignment | Scene graph generation · Action Genome | F@20/50: 25.2 / 29.1 -> 25.5 / 29.7 | Helped |
| add reciprocal refinement between aligned triplet and component features (ablation) | Hybrid reciprocal transformer · Visual Genome | mR@50: 14.4 -> 16.0 | Helped |
| add triplet-level guidance for component-level learning (ablation) | Hybrid reciprocal transformer · Visual Genome | R@50: 33.2 -> 34.1 | Helped |
| add triplet feature alignment loss with weight 0.2 rather than weight 0.0 (ablation) | Hybrid reciprocal transformer · Visual Genome | R@50: 33.8 -> 34.1 | Helped |
| increase triplet alignment loss weight from 0.2 to 1.0 (ablation) | Hybrid reciprocal transformer · Visual Genome | R@50: 34.1 -> 32.4 | Hurt |
| increase triplet alignment loss weight from 0.2 to 1.0 (ablation) | Hybrid reciprocal transformer · Visual Genome | mR@100: 20.5 -> 19.0 | Hurt |
Surprising findings
- Directly multiplying triplet-level and component-level scores at inference degrades performance (R@50 32.5 vs 34.1), while Hungarian-matched multiplication gives only a marginal gain.
- Conditioning triplet queries on component queries (reversed guidance order) performs worse than the proposed triplet-to-component order.
Limitations
- Alignment loss weight has a narrow useful range; higher weights degrade performance.
- Grouping hybrid representations under one uniform Hungarian matcher reduces performance.
Experimental setup
Datasets: Visual Genome Action Genome
Benchmarks: Visual Genome Action Genome
Metrics: R@K mR@K AvgR@K F@K
Baselines: ISG SpeaQISG DRM PE-Net Hydra-SGG EGTR RelTR SGTR
Builds on: SpeaQ ISG Mask2Former Deformable DETR
Closest papers by meaning
- Leveraging Predicate and Triplet Learning for Scene Graph Generation (CVPR 2024 · similarity 0.85)
- TrioPose: Native Triple-Stream Diffusion Transformers for Pose-Guided Text-to-Image Generation (NeurIPS 2026 · similarity 0.84)
- LLM4SGG: Large Language Models for Weakly Supervised Scene Graph Generation (CVPR 2024 · similarity 0.83)
- DOCTR: Disentangled Object-Centric Transformer for Point Scene Understanding (AAAI 2024 · similarity 0.83)
- VSFormer: Visual-Spatial Fusion Transformer for Correspondence Pruning (AAAI 2024 · similarity 0.83)
- SIU3R: Simultaneous Scene Understanding and 3D Reconstruction Beyond Feature Alignment (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.