Research map / Video / Video-language understanding
Weakly Supervised Video Scene Graph Generation via Natural Language Supervision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NL-VSGG, which trains a video scene graph generation model using only video captions by LLM-based caption segmentation, variability-aware caption-frame clustering alignment, and motion-cue pseudo-labeling of negative classes.
Problem
Video scene graph generation (VidSGG) requires costly frame-by-frame annotations; naively transferring weakly supervised image SGG pipelines using captions fails because video captions contain temporal markers and human action durations vary.
Approach
Uses only video captions as weak supervision: an LLM segments captions into temporally ordered sentences, K-means clustering of frame features with a steepest-decline similarity cut aligns each sentence to variable-length frame spans, and GIoU-based motion cues pseudo-label negative action classes in unaligned frames.
Key results
- On Action Genome with STTran, NL-VSGG reaches R@50 19.60 (with constraint) and 22.56 (no constraint), versus 12.83 and 14.05 for the naive WS-ImgSGG pipeline.
- Ablations show TCS, ADV, and PLM each add gains; PLM contributes most on the AG negative classes (mean R@50 rises from 16.47 to 21.08 with PLM).
- Training on AG+MSVD captions improves With Constraint R@50 to 20.00 versus 19.60 for AG alone; NL-VSGG remains below fully supervised STTran (36.93 R@50).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use NL-VSGG instead of WS-ImgSGG | STTran · Action Genome | With Constraint R@20: 10.01 -> 15.61 | Helped |
| use NL-VSGG instead of PLA_cap | STTran · Action Genome | With Constraint R@20: 10.40 -> 15.61 | Helped |
| add TCS to the WS-ImgSGG pipeline (ablation) | STTran on Action Genome · Action Genome | With Constraint R@20: 10.01 -> 11.09 | Helped |
| add ADV to TCS (ablation) | STTran on Action Genome · Action Genome | With Constraint R@20: 11.09 -> 11.98 | Helped |
| add PLM to TCS + ADV (ablation) | STTran on Action Genome · Action Genome | With Constraint R@20: 11.98 -> 15.61 | Helped |
| add TCS to the WS-ImgSGG pipeline (ablation) | STTran on Action Genome without negative classes · Action Genome without negative classes | With Constraint R@20: 12.98 -> 15.34 | Helped |
| add ADV to TCS (ablation) | STTran on Action Genome without negative classes · Action Genome without negative classes | With Constraint R@20: 15.34 -> 15.95 | Helped |
| include the coreference-resolution prompt (ablation) | NL-VSGG with STTran · Action Genome | With Constraint R@20: 15.48 -> 15.61 | Helped |
| replace ChatGPT with Mistral-7B in TCS (ablation) | NL-VSGG with STTran · Action Genome | With Constraint R@20: 15.61 -> 14.58 | Hurt |
| replace DAC with InternVideo in ADV (ablation) | NL-VSGG with STTran · Action Genome | With Constraint R@20: 15.61 -> 15.58 | No clear effect |
| replace LLM-based parsing with SG Parser + KB (ablation) | NL-VSGG with STTran · Action Genome | With Constraint R@20: 15.61 -> 11.08 | Hurt |
| add MSVD captions to AG caption training data | NL-VSGG with STTran · Action Genome | With Constraint R@20: 15.61 -> 15.71 | Helped |
Surprising findings
- Adding PLA's middle-frame GT scene graph is much stronger than captions, but a naive PLA with caption-derived graphs (PLA_cap) barely improves over WS-ImgSGG.
- Replacing DAC with the video-language model InternVideo gives competitive results, suggesting frame averaging in clustering suffices for temporal context.
Limitations
- Performance remains well below fully supervised VidSGG models (e.g., STTran full R@50 36.93 vs 19.60).
- Limitation regarding untrimmed, long-duration videos noted by the authors.
- Relies on an object detector and LLM parsing whose errors (grounding failures, triplet parsing) propagate to supervision.
Experimental setup
Datasets: Action Genome caption MSVD caption ActivityNet caption Action Genome VidHOI
Benchmarks: Action Genome VidHOI
Metrics: Recall@20 Recall@50 (with constraint) Recall@50 (no constraint)
Baselines: RLIP RLIPv2 STTran (full supervision) DSG-DETR (full supervision) PLA PLA_simp. WS-ImgSGG PLA_cap
Builds on: PLA STTran DSG-DETR WS-ImgSGG pipeline (SGNLS/LSWS) DAC VinVL
Closest papers by meaning
- A Closer Look at Dynamic Scene Graph Generation in the Era of Multimodal Large Language Models (NeurIPS 2026 · similarity 0.89)
- LLM4SGG: Large Language Models for Weakly Supervised Scene Graph Generation (CVPR 2024 · similarity 0.89)
- LASER: A Neuro-Symbolic Framework for Learning Spatio-Temporal Scene Graphs with Weak Supervision (ICLR 2025 · similarity 0.89)
- Fine-Grained Captioning of Long Videos through Scene Graph Consolidation (ICML 2025 · similarity 0.88)
- DiffVsgg: Diffusion-Driven Online Video Scene Graph Generation (CVPR 2025 · similarity 0.87)
- OED: Towards One-stage End-to-End Dynamic Scene Graph Generation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.