Research map / Vision-language and multimodal / Human and embodied modeling
From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frames open-vocabulary scene graph generation as image-to-sequence generation by fine-tuning BLIP to emit scene-graph token sequences, then grounding entities with a box head and mapping vocabulary tokens to category labels.
Problem
Open-vocabulary scene graph generation usually covers only novel entities or classifies predicates given ground-truth pairs; generating scene graphs with novel predicates end-to-end from pixels is under-explored, and prior methods need costly VLM pre-training.
Approach
Use a generative VLM (BLIP) to output scene graphs as text triplets with relation-aware [ENT]/[REL] tokens, ground entities with a small transformer box-regression head over the image features, and convert vocabulary token scores to benchmark categories without extra parameters; the fine-tuned weights then initialize VL tasks.
Key results
- On VG SGCls, improves base+novel mR@100 and R@100 over SVRP by 6.5 and 5.7 points.
- On PSG, gains of 9.4 mR@100 and 14.3 R@100 over ViT-based SGTR in open-vocabulary SGDet.
- On GQA, PGSG improves BLIP fine-tuned overall accuracy from 62.5 to 64.2 (per-table result), and zero-shot BLIP v2 from 32.3 to 33.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use image-to-sequence scene graph generation | PGSG · VG SGDet, Novel+base | mR@100: 5.4 -> 8.3 | Helped |
| use image-to-sequence scene graph generation | PGSG · PSG, Novel+base | mR@100: 8.4 -> 18.1 | Helped |
| use image-to-sequence scene graph generation | PGSG · OpenImages-V6, Novel+base | mR@100: 16.7 -> 23.0 | Helped |
| initialize VL-task fine-tuning from the SGG-trained PGSG model | BLIP · RefCOCO val | visual grounding: 83.1 -> 86.0 | Helped |
| initialize VL-task fine-tuning from the SGG-trained PGSG model | BLIP · GQA | overall accuracy: 62.5 -> 64.2 | Helped |
| use 6 transformer layers rather than 0 in entity grounding (ablation) | PGSG · PSG SGDet | mR@100: 10.3 -> 17.0 | Helped |
| use relation-aware tokens rather than mixture for entity grounding (ablation) | PGSG · PSG SGDet | mR@100: 8.5 -> 17.0 | Helped |
| set output sequence length to 256 rather than 1024 (ablation) | PGSG · PSG open-vocabulary SGG | mR@100: 25.0 -> 18.3 | Hurt |
| set output sequence length to 768 rather than 1024 (ablation) | PGSG · PSG open-vocabulary SGG | mR@50: 22.9 -> 23.6 | Helped |
| fine-tune with entity-only sequence prompts instead of scene graph supervision (ablation) | PGSG-E · GQA | overall accuracy: 64.2 -> 63.1 | Hurt |
| fine-tune BLIP on COCO image captioning from its initial model | BLIP · COCO image captioning | SPICE: 23.6 -> 23.9 | Helped |
Surprising findings
- Halving the generated sequence length keeps performance comparable while reducing inference time (6.9s to 2.2s per image at 512 tokens).
- Close-vocabulary SGG performance lags prior SOTA (e.g., ResNet-101 based methods) due to low input resolution (384x384) and single-stage training.
Limitations
- Close-vocabulary SGG performance remains below SOTA due to weak visual backbone at low resolution and labeling noise.
- Only BLIP/BLIP v2 and three VL tasks studied; generalization to other VLMs is untested.
- Heuristic rule-based triplet parsing from generated sequences.
Experimental setup
Datasets: Panoptic Scene Graph (PSG) Visual Genome (VG) OpenImages V6 RefCOCO RefCOCO+ RefCOCOg GQA COCO
Benchmarks: PSG Visual Genome OpenImages V6 RefCOCO RefCOCO+ RefCOCOg GQA COCO captioning
Metrics: R@K mR@K zero-shot recall zR@K BLEU4 CIDEr SPICE accuracy
Baselines: SVRP CaCao VS3 SGTR PSGTR PSGFormer RelTR ISG SSRCNN
Builds on: BLIP InstructBLIP SGTR nucleus sampling
Closest papers by meaning
- Relation-aware Hierarchical Prompt for Open-vocabulary Scene Graph Generation (AAAI 2025 · similarity 0.92)
- Scene Graph Generation with Role-Playing Large Language Models (NeurIPS 2024 · similarity 0.91)
- A Closer Look at Dynamic Scene Graph Generation in the Era of Multimodal Large Language Models (NeurIPS 2026 · similarity 0.90)
- Scene Graph-Grounded Image Generation (AAAI 2025 · similarity 0.89)
- The Scene Language: Representing Scenes with Programs, Words, and Embeddings (CVPR 2025 · similarity 0.89)
- Leveraging Predicate and Triplet Learning for Scene Graph Generation (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.