Research map / Vision-language and multimodal / Human and embodied modeling
Relation-aware Hierarchical Prompt for Open-vocabulary Scene Graph Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RAHP, a relation-aware hierarchical prompt method for open-vocabulary scene graph generation that combines entity-aware and region-aware text prompts with a VLM distillation loss, evaluated on Visual Genome and Open Image V6.
Problem
Open-vocabulary scene graph generation relies on fixed text representations of relationships, which limits the ability to recognize novel visual relationships and align visual and textual relation representations.
Approach
Builds hierarchical text prompts: entities are clustered into super entities (via WordNet POS, VLM embeddings, K-means, and LLM naming) to form entity-aware prompts, and an LLM generates multiple region-aware descriptions per triplet, dynamically selecting the top-k for predicate scoring, with a distillation loss aligning SGG relation features with VLM visual features.
Key results
- On VG OVR-SGG PredCLS with PE-NET, RAHP raises total R@100 from 61.23 to 69.11 and novel-predicate R@100 from 0.00 to 29.00.
- Zero-shot predicate classification with CLIP ViT-B/32: RAHP improves R@20 from 10.60 to 11.06 and mR@20 from 10.70 to 12.48 over RECODE.
- Close-vocabulary SGDet on VG: PE-NET+RAHP improves R@100 from 35.20 to 36.46 over PE-NET.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace PE-NET's fixed classifier with RAHP's open-vocabulary classifier | PE-NET · VG SGDet close-vocabulary | R@50/100: 30.70/35.20 -> 31.52/36.46 | Helped |
| replace VS$^3$'s fixed classifier with RAHP's open-vocabulary classifier | VS$^3$ · VG SGDet close-vocabulary | R@50/100: 35.80/41.30 -> 34.25/40.40 | Hurt |
| replace SGTR$^\dagger$'s fixed classifier with RAHP's open-vocabulary classifier | SGTR$^\dagger$ · VG SGDet close-vocabulary | mR@50/100: 12.60/17.00 -> 12.49/17.04 | Mixed |
| add RAHP prompts instead of using fixed-format text prompts | SGTR$^\dagger$ · VG SGCLS test, total relations | R@50/100: 25.98/29.31 -> 26.65/30.83 | Helped |
| add RAHP prompts instead of using fixed-format text prompts | SGTR$^\dagger$ · VG SGCLS test, novel relations | R@50/100: 11.03/18.52 -> 14.37/21.72 | Helped |
| use GPT-4o-mini instead of GPT-3.5-turbo for prompt generation (ablation) | PE-NET+RAHP · VG validation PredCLS, total relations | R@50/100: 64.70/69.11 -> 64.73/69.15 | No clear effect |
| use RAHP regional-level descriptions instead of RECODE | CLIP zero-shot visual relationship detection · VG | R@20: 10.60 -> 11.06 | Helped |
| use RAHP regional-level descriptions instead of RECODE | CLIP zero-shot visual relationship detection · VG | mR@20: 10.70 -> 12.48 | Helped |
| set region-aware prompt weight α to 0 instead of the best reported α=0.25 (ablation) | RAHP with PE-NET · VG validation, base and novel predicates | performance | Hurt |
| set region-aware prompt weight α to 1 instead of the best reported α=0.25 (ablation) | RAHP with PE-NET · VG validation, base and novel predicates | performance | Hurt |
| select all region-aware prompts instead of k=3 (ablation) | RAHP · VG validation, novel predicates | R@100 | Hurt |
| select only the top-scoring prompt instead of k=3 (ablation) | RAHP · VG validation | R@100 | Hurt |
Surprising findings
- Replacing the fixed classifier with RAHP in VS^3 lowers mR@50/100 notably (7.21/10.45), so gains are not uniform across baselines.
Limitations
- Relies on LLM-generated region descriptions whose generation cost is noted as not filled in (placeholder '[insert cost here]').
- Fewer super entities lose entity-specific information, so granularity trades off against completeness.
Experimental setup
Datasets: Visual Genome Open Image V6
Benchmarks: Visual Genome Open Image V6
Metrics: R@50 R@100 mR@50 mR@100
Baselines: PE-NET VS^3 SGTR PGSG OvSGTR RECODE
Builds on: PE-NET OvSGTR RECODE
Closest papers by meaning
- Scene Graph Generation with Role-Playing Large Language Models (NeurIPS 2024 · similarity 0.92)
- From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models (CVPR 2024 · similarity 0.92)
- Interaction-Centric Knowledge Infusion and Transfer for Open Vocabulary Scene Graph Generation (NeurIPS 2025 · similarity 0.89)
- A Closer Look at Dynamic Scene Graph Generation in the Era of Multimodal Large Language Models (NeurIPS 2026 · similarity 0.89)
- Leveraging Predicate and Triplet Learning for Scene Graph Generation (CVPR 2024 · similarity 0.88)
- Compositional Chain-of-Thought Prompting for Large Multimodal Models (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.