Research map / Vision-language and multimodal / Human and embodied modeling
LLM4SGG: Large Language Models for Weakly Supervised Scene Graph Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses an LLM (ChatGPT) with chain-of-thought-style prompts and few-shot examples to replace the rule-based parser and KB alignment in weakly-supervised scene graph generation, mitigating semantic over-simplification and low-density scene graphs.
Problem
Weakly-supervised scene graph generation from image captions suffers from semantic over-simplification of predicates by rule-based parsers and low-density scene graphs because knowledge-base alignment discards many triplets.
Approach
Replace the rule-based scene graph parser and WordNet-based class alignment with an LLM prompted in a stepwise (chain-of-thought-like) manner with in-context examples, and add triplets from a paraphrased caption to densify the scene graph; the LLM is used without fine-tuning.
Key results
- On Visual Genome, VS3+LLM4SGG (COCO captions) reaches R@100 10.43 and mR@100 8.18 versus VS3's 8.01 and 3.25 under SGDet.
- On GQA, VS3+LLM4SGG improves R@100 from 6.97 to 10.38 and mR@100 from 1.81 to 6.51 over VS3.
- Data-efficiency: with only 5K training images (7.8%), VS3+LLM4SGG outperforms VS3 trained on 64K images.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply LLM4SGG triplet extraction and alignment | VS3 · VG, COCO Caption | mR@50: 2.88 -> 7.11 | Helped |
| Apply LLM4SGG triplet extraction and alignment | SGNLS · VG, COCO Caption | mR@50: 2.51 -> 4.08 | Helped |
| Apply LLM4SGG triplet extraction and alignment | VS3 · VG, CC Caption | mR@50: 1.73 -> 5.40 | Helped |
| Apply LLM4SGG triplet extraction and alignment | VS3 · VG, VG Caption | mR@50: 2.80 -> 6.26 | Helped |
| Apply LLM4SGG triplet extraction and alignment | VS3 · GQA | mR@50: 1.60 -> 5.33 | Helped |
| Add paraphrased captions to the original captions (ablation) | VS3 · VG, COCO Caption | R@50: 6.60 -> 9.46 | Helped |
| Add paraphrased captions to the original captions (ablation) | VS3 · VG, COCO Caption | mR@50: 2.88 -> 3.43 | Helped |
| Replace conventional parsing with LLM-based parsing (ablation) | VS3 with paraphrased captions · VG, COCO Caption | mR@50: 3.43 -> 5.99 | Helped |
| Replace conventional parsing with LLM-based parsing (ablation) | VS3 with paraphrased captions · VG, COCO Caption | R@50: 9.46 -> 8.42 | Hurt |
| Replace conventional triplet alignment with LLM-based alignment (ablation) | VS3 with paraphrased captions · VG, COCO Caption | R@50: 9.46 -> 11.76 | Helped |
| Replace conventional triplet alignment with LLM-based alignment (ablation) | VS3 with paraphrased captions · VG, COCO Caption | mR@50: 3.43 -> 3.50 | Helped |
| Combine LLM-based parsing, LLM-based alignment, and paraphrased captions (ablation) | VS3 · VG, COCO Caption | F@50: 4.01 -> 7.91 | Helped |
Surprising findings
- Under the conventional parser+KB pipeline, 12 of 50 Visual Genome predicates have zero frequency in unlocalized triplets.
- SGNLS grounding fails on about 29% (100K of 344K) of LLM-generated unlocalized triplets, limiting gains for that grounding method.
Limitations
- Relies on a proprietary black-box LLM (ChatGPT) that can be costly to use.
- Only a single prompt design was tested.
Experimental setup
Datasets: COCO caption Conceptual Captions Visual Genome GQA
Benchmarks: Visual Genome GQA
Metrics: R@K mR@K F@K
Baselines: LSWS SGNLS Li et al (MM'22) VS3 VS3+Rwt Motif
Builds on: Chain-of-Thought VS3 SGNLS in-context learning
Closest papers by meaning
- Leveraging Predicate and Triplet Learning for Scene Graph Generation (CVPR 2024 · similarity 0.90)
- Weakly Supervised Video Scene Graph Generation via Natural Language Supervision (ICLR 2025 · similarity 0.89)
- From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models (CVPR 2024 · similarity 0.89)
- RA-SGG: Retrieval-Augmented Scene Graph Generation Framework via Multi-Prototype Learning (AAAI 2025 · similarity 0.89)
- Scene Graph Generation with Role-Playing Large Language Models (NeurIPS 2024 · similarity 0.89)
- A Closer Look at Dynamic Scene Graph Generation in the Era of Multimodal Large Language Models (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.