Research map / Detection and segmentation / Open-set and few-shot detection
Interaction-Centric Knowledge Infusion and Transfer for Open Vocabulary Scene Graph Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ACC, an end-to-end interaction-centric open-vocabulary scene graph generation framework that adds bidirectional interaction prompts for pseudo-supervision, interaction-guided query selection, and interaction-consistent distillation.
Problem
Open-vocabulary scene graph generation methods treat instances of the same object category uniformly, causing noisy pseudo-supervision during knowledge infusion (pre-training) and ambiguous query-to-ground-truth matching during knowledge transfer (SFT).
Approach
Makes both knowledge infusion and transfer interaction-centric: bidirectional interaction prompts (subject-view and counter-action object-view) guide grounding for pseudo-triplets; two-step interaction-guided query selection prioritizes interacting objects; and interaction-consistent KD combines point-wise edge alignment with inter-pair structural similarity alignment.
Key results
- On VG OvD+R-SGG with Swin-B, ACC reaches 23.19 R@100 joint base+novel vs. OvSGTR's 21.03.
- On VG OvR-SGG with Swin-B, ACC achieves 29.28 R@100 on base+novel relations and 24.66 on novel relations.
- Ablation: IGQS gives about +3.00 R@100 and ICKD about +2.83 R@100 over the baseline in the OvD+R-SGG setting.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use ACC instead of OvSGTR | VG OvR-SGG · VG test, OvR-SGG Novel (Relation), Swin-T | R@100: 19.92 -> 21.70 | Helped |
| add bidirectional interaction prompt for pseudo-supervision generation (ablation) | ACC · VG test, OvD+R-SGG Joint Base+Novel | R@100: 19.55 -> 21.27 | Helped |
| add bidirectional interaction prompt for pseudo-supervision generation (ablation) | ACC · VG test, OvD+R-SGG Novel (Obj) | R@100: 19.65 -> 21.10 | Helped |
| add bidirectional interaction prompt for pseudo-supervision generation (ablation) | ACC · VG test, OvD+R-SGG Novel (Rel) | R@100: 17.83 -> 19.46 | Helped |
| add Step I interaction/object-relation relevance query selection (ablation) | ACC knowledge transfer · VG test, OvD+R-SGG Joint Base+Novel | R@100: 16.37 -> 19.16 | Helped |
| add Step II interaction-guided query selection (ablation) | ACC knowledge transfer · VG test, OvD+R-SGG Joint Base+Novel | R@100: 16.37 -> 19.29 | Helped |
| use both query-selection steps instead of neither (ablation) | ACC knowledge transfer · VG test, OvD+R-SGG Joint Base+Novel | R@100: 16.37 -> 19.37 | Helped |
| replace Llama2-7B verb parser with Qwen2.5-0.5B (ablation) | ACC bidirectional interaction prompt · VG test, OvD+R-SGG Joint Base+Novel | R@100: 23.19 -> 23.43 | Helped |
| replace Llama2-7B verb parser with Pattern Python library (ablation) | ACC bidirectional interaction prompt · VG test, OvD+R-SGG Joint Base+Novel | R@100: 23.19 -> 22.64 | Hurt |
Surprising findings
- Combining IGQS and ICKD yields smaller gains than the sum of individual gains, which the authors attribute to diminishing returns from reducing non-interacting objects.
Limitations
- Step II requires an extra forward pass, increasing inference cost (0.64 s/image vs 0.39 s).
- Gains from combining the two transfer components are smaller than expected due to diminishing returns.
Experimental setup
Datasets: Visual Genome (VG150) GQA200 PSG COCO captions HICO-DET
Benchmarks: VG150 (OvR-SGG, OvD+R-SGG) GQA200 (OvD+R-SGG) PSG (OvR-SGG) HICO-DET (OvR-SGG)
Metrics: R@20 R@50 R@100 mR@K
Baselines: IMP MOTIFS VCTREE TDE OpenSGen VS^3 OvSGTR RAHP SGTR PGSG LSWS Uniter
Builds on: Grounding DINO OvSGTR DETR visual-concept retention distillation
Closest papers by meaning
- Relation-aware Hierarchical Prompt for Open-vocabulary Scene Graph Generation (AAAI 2025 · similarity 0.89)
- Scene Graph Generation with Role-Playing Large Language Models (NeurIPS 2024 · similarity 0.89)
- From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models (CVPR 2024 · similarity 0.87)
- Synthetic Visual Genome (CVPR 2025 · similarity 0.86)
- Leveraging Predicate and Triplet Learning for Scene Graph Generation (CVPR 2024 · similarity 0.86)
- A Closer Look at Dynamic Scene Graph Generation in the Era of Multimodal Large Language Models (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.