Research map / Vision-language and multimodal / Human and embodied modeling
Synthetic Visual Genome
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds the SVG dataset by using GPT-4V to complete missing relations on seed scene graphs, trains ROBIN-3B (Qwen2.5-3B based, Osprey-style) on it, and refines its own dense scene graphs with GPT-4o in a self-distillation loop (SG-EDIT).
Problem
Open-source multimodal language models struggle with precise visual relationship reasoning and dense scene graph generation, largely because large-scale relationship-focused instruction data is lacking and sparse human annotations like Visual Genome cover few relations per object.
Approach
Complete missing relations on seed scene-graph objects with a teacher MLM plus rule and VQA filtering, instruction-tune a grounded MLM on the result, then iteratively refine the model's own dense scene graphs with GPT-4o editing and retrain (self-distillation).
Key results
- ROBIN-3B reaches 88.2% average referring expression comprehension accuracy, versus 87.4 for the previous best (ASM-V2-13B 87.3 in its own table).
- On What's Up (Controlled), ROBIN-3B scores 80.1% vs 78.7% for Phi-3-Vision and 78.2% for BLIP-3, and it improves CRPE relation score from 64.8% (Stage 1) to 68.0% (Stage 2).
- On PSG open-ended scene graph generation, ROBIN-3B reaches R@20 of 20.6 and mR@20 of 13.2, above ASM-V2-13B (14.2/10.3).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add SG-EDIT training with GPT-4 edited scene graphs (ablation) | Robin-3B Stage 1 · CRPE Relation | accuracy: 64.8% vs 68.0% | Helped |
| add SG-EDIT training with GPT-4 edited scene graphs (ablation) | Robin-3B Stage 1 · SugarCrepe Relation | accuracy: 87.9% vs 89.4% | Helped |
| add SG-EDIT training with GPT-4 edited scene graphs (ablation) | Robin-3B Stage 1 · What's Up? Controlled | accuracy: 76.7% vs 80.1% | Helped |
| add SG-EDIT training with GPT-4 edited scene graphs (ablation) | Robin-3B Stage 1 · MMBench Image | score: 77.1 -> 77.0 | Hurt |
| add SG-EDIT training with GPT-4 edited scene graphs (ablation) | Robin-3B Stage 1 · RefCOCO/RefCOCO+/RefCOCOg test average | Recall@1: 86.0% → 88.2% | Helped |
| add SG-EDIT training with GPT-4 edited scene graphs (ablation) | Robin-3B Stage 1 · PSG | R@20: 18.2 -> 20.6 | Helped |
| add scene graph data to visual instruction and grounding data (ablation) | Stage 1 training mixture · VSR | accuracy: 68.2 -> 69.7 | Helped |
| add scene graph data to visual instruction and grounding data (ablation) | Stage 1 training mixture · CRPE Relation | accuracy: 64.1 -> 64.8 | Helped |
| add scene graph data to visual instruction data without grounding data (ablation) | Stage 1 training mixture · PACO | SS: 68.9 -> 72.5 | Helped |
| train on model-generated scene graphs without GPT-4 editing (ablation) | Robin-3B Stage 1 · VSR | accuracy: -0.9% | Hurt |
| train on model-generated scene graphs without GPT-4 editing (ablation) | Robin-3B Stage 1 · What's Up? | accuracy: -1.0% | Hurt |
| replace model-generated scene graphs with GPT-4 edited scene graphs (ablation) | Robin-3B Stage 1 · relationship understanding benchmarks | average performance: 2.2% | Helped |
Surprising findings
- Model-generated scene graphs appended as context outperform GPT-4 edits of human-annotated Visual Genome graphs in blind-LLM VQA, suggesting the model captures relations missing from VG.
Limitations
- Scene graph generation is evaluated only against existing COCO-based annotations, not on in-the-wild images.
- Synthetic scene graphs depend on GPT-4V/GPT-4o quality and may contain hallucinations; the paper notes frontier models hallucinate when generating from scratch.
- SVG-FULL was limited to 93-113K scene graphs; scaling behavior is not explored.
Experimental setup
Datasets: COCO LVIS RefCOCO Visual Genome GQA Depth-Anything PSG ADE20K LLaVA-Pretrain-558K Osprey-724K
Benchmarks: GQA MMBench SEED-Bench VSR CRPE SugarCrepe What's Up RefCOCO RefCOCO+ RefCOCOg ADE20K LVIS
Metrics: accuracy Recall@1 PQ mAP mIoU semantic similarity S-IOU R@20 mR@20
Baselines: VILA1.5-3B MiniCPM-V2.0-3B InternVL2-2B Phi-3-Vision BLIP-3 MM1.5-3B ASM-V2-13B Osprey-7B LLaVA-1.5-13B VisionLLMv2-Chat
Builds on: Osprey Visual Genome Segment Anything LLaVA
Closest papers by meaning
- Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models (ICML 2026 · similarity 0.90)
- Compositional Chain-of-Thought Prompting for Large Multimodal Models (CVPR 2024 · similarity 0.88)
- Leveraging Predicate and Triplet Learning for Scene Graph Generation (CVPR 2024 · similarity 0.88)
- From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models (CVPR 2024 · similarity 0.88)
- HyperGLM: HyperGraph for Video Scene Graph Generation and Anticipation (CVPR 2025 · similarity 0.88)
- Panoptic Scene Graph Generation with Semantics-Prototype Learning (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.