Research map / Vision-language and multimodal / CLIP and zero-shot
Scene Graph Generation with Role-Playing Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SDSGG, an open-vocabulary scene graph generation framework that replaces fixed category/part-level text classifiers with LLM-generated scene-specific descriptions, renormalized by opposite descriptions, plus a mutual visual adapter over CLIP features.
Problem
Open-vocabulary scene graph generation with CLIP relies on scene-agnostic text classifiers (category names or part-level prompts), which struggle with high-variance visual relations and can be misleading across contexts.
Approach
Generate scene-specific descriptions (SSDs) via LLM multi-persona collaboration, pair each with an LLM-generated opposite description, and score relations by the difference of their CLIP similarities (self-normalized similarity), together with a lightweight cross-attention mutual visual adapter that models subject-object interplay.
Key results
- On VG novel split, SDSGG reaches 25.4 R@50 and 29.6 R@100, exceeding Epic by 18.0/19.9 points.
- On VG semantic split, SDSGG exceeds RECODE* by 10.9/11.0/9.9 points on R@20/50/100.
- Removing multi-persona collaboration drops VG base R@100 from 31.6 to 11.8.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SDSGG with scene-specific descriptions and its full method instead of Epic | OVSGG on VG base and novel · VG base and novel | R@100: 27.2 -> 31.6; 9.7 -> 29.6 | Mixed |
| Use SDSGG instead of CLS | OVSGG on GQA · GQA base and novel | R@100: 7.9 -> 49.5; 32.1 -> 42.9 | Helped |
| Add multi-persona collaboration for description generation (ablation) | SDSGG · VG base and novel | R@100: 11.8 -> 31.6; 11.8 -> 30.0 | Helped |
| Add the mutual visual adapter with cross-attention instead of the MLP baseline (ablation) | SDSGG · VG base and novel | R@100: 22.4 -> 31.6; 28.9 -> 30.0 | Helped |
| Add the directional marker to MVA (ablation) | SDSGG · VG base and novel | R@100: 28.2 -> 31.6; 28.7 -> 30.0 | Helped |
| Add opposite-description reference for self-normalized similarity (ablation) | SDSGG · VG base and novel | R@100: 28.3 -> 31.6; 25.5 -> 30.0 | Helped |
| Increase the number of SSD pairs from 11 to 21 (ablation) | SDSGG · VG base and novel | R@100: 24.0 -> 31.6; 26.8 -> 30.0 | Helped |
| Increase the number of SSD pairs from 21 to 26 (ablation) | SDSGG · VG base and novel | R@100: 31.6 -> 20.0; 30.0 -> 23.7 | Hurt |
| Change MVA attention heads from 4 to 8 (ablation) | SDSGG · VG semantic | R@100: 32.7 -> 35.1 | Helped |
| Change MVA attention heads from 8 to 16 (ablation) | SDSGG · VG base and novel | R@100: 31.6 -> 32.8; 30.0 -> 29.0 | Mixed |
| Vary training scaling factors alpha and beta among the tested values (ablation) | SDSGG · VG base and novel | R@100 | No clear effect |
| Vary training margin lambda among the tested values (ablation) | SDSGG · VG base and novel | R@100 | No clear effect |
Surprising findings
- Single-persona descriptions (biologist, engineer, physicist) perform far worse than multi-persona collaboration, with mR@50 on base falling from 12.3 to 4.8-7.3.
Limitations
- Designed only for predicate classification with ground-truth object boxes, not end-to-end SGG.
- Relies on LLM-generated descriptions whose completeness and quality are not guaranteed.
- Potential bias inherited from large-scale VLM and LLM pretraining.
Experimental setup
Datasets: Visual Genome GQA
Benchmarks: Visual Genome GQA
Metrics: R@K mR@K
Baselines: CLS Epic CLSDE RECODE RECODE*
Builds on: CLIP RECODE CLS (category-level prompts)
Closest papers by meaning
- Relation-aware Hierarchical Prompt for Open-vocabulary Scene Graph Generation (AAAI 2025 · similarity 0.92)
- From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models (CVPR 2024 · similarity 0.91)
- A Closer Look at Dynamic Scene Graph Generation in the Era of Multimodal Large Language Models (NeurIPS 2026 · similarity 0.90)
- Interaction-Centric Knowledge Infusion and Transfer for Open Vocabulary Scene Graph Generation (NeurIPS 2025 · similarity 0.89)
- The Scene Language: Representing Scenes with Programs, Words, and Embeddings (CVPR 2025 · similarity 0.89)
- Universal Scene Graph Generation (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.