Research map / Video / Video-language understanding
LASER: A Neuro-Symbolic Framework for Learning Spatio-Temporal Scene Graphs with Weak Supervision
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LASER, a neuro-symbolic framework that trains spatio-temporal scene graph generators from video captions alone, using LLM-extracted LTLf-style specifications and a differentiable Scallop-based alignment checker with contrastive, temporal, and semantic losses.
Problem
Learning spatio-temporal scene graphs (STSG) from video normally requires costly STSG annotations; the paper asks whether readily available video captions can serve as weak supervision despite their sparsity and noise.
Approach
Convert captions into logical spatio-temporal specifications (STSL, based on LTLf) with GPT-4, then train any STSG generator end-to-end by maximizing the probability that predicted probabilistic scene graphs satisfy the specification, computed with a differentiable Scallop reasoner and augmented by contrastive, time-span, and semantic (integrity-constraint) losses.
Key results
- On OpenPVSG, LASER-tuned CLIP reaches unary R@1 0.2778 (+0.1145 over base) and binary R@5 0.4214 (+0.354 over base), with weak supervision from captions only.
- On 20BN, LASER achieves F1 0.73 and 83% accuracy versus GPA's 0.74 F1 and 76% accuracy; LASER-P (ground-truth specs) reaches 0.77 F1 and 91% accuracy.
- On MUGEN, LASER with 5,000 training points reaches 59.6% action accuracy versus 54.4% for the fully supervised baseline; with 10% of OpenPVSG data it achieves about 70.75% of full-data gains.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune CLIP with LASER weak supervision from captions | CLIP · OpenPVSG | Unary R@1: 0.1633 -> 0.2778 | Helped |
| Fine-tune CLIP with LASER weak supervision from captions | CLIP · OpenPVSG | Binary R@5: 0.0673 -> 0.4214 | Helped |
| Fine-tune VIOLET with LASER weak supervision from captions | VIOLET · OpenPVSG | Unary R@1: 0.0660 -> 0.0878 | Helped |
| Fine-tune SigLIP with LASER weak supervision from captions | SigLIP · OpenPVSG | Unary R@1: 0.0000 -> 0.1467 | Helped |
| Use caption-generated specifications instead of GPA supervision | 20BN STSG generator · 20BN | accuracy: 76% -> 83% | Helped |
| Use caption-generated specifications instead of ground-truth programmatic specifications | 20BN STSG generator · 20BN | F1: 0.77 -> 0.73 | Hurt |
| Add temporal loss (ablation) | Full LASER loss · 20BN | F1: 0.31 -> 0.77 | Helped |
| Add semantic loss (ablation) | Contrastive and temporal loss · 20BN | F1: 0.58 -> 0.77 | Helped |
| Add semantic loss while using contrastive loss only (ablation) | 20BN loss setup · 20BN | F1: 0.31 -> 0.31 | No clear effect |
| Train on 5,000 data points with caption-derived specifications instead of direct supervision | MUGEN action model · MUGEN (5,000 data points) | action prediction accuracy: 54.4 -> 59.6 | Helped |
| Train CLIP with LASER on LLaVA-Video captions rather than use the base CLIP model | CLIP · OpenPVSG | Unary R@1: 0.1633 -> 0.2368 | Helped |
| Use generated STSL directly without fine-tuning CLIP | OpenPVSG STSG prediction · OpenPVSG | Unary R@1: 0.2778 -> 0.0514 | Hurt |
Surprising findings
- Weakly supervised LASER outperforms fully supervised baselines on OpenPVSG and MUGEN.
- Directly using caption-extracted STSL as a scene graph (STSL-as-STSG) performs far worse than fine-tuned CLIP, due to mismatched granularity.
Limitations
- Scalability challenges with long videos and many objects per frame.
- Performance bounded by quality of video captions.
- Relies on GPT-4 for caption-to-specification conversion; 20BN and MUGEN use ground-truth-like or heuristic specifications.
Experimental setup
Datasets: OpenPVSG 20BN-Something-Something MUGEN LLaVA-Video 10K Action Genome VidVRD
Benchmarks: OpenPVSG 20BN-Something-Something MUGEN Action Genome
Metrics: unary R@k binary R@k F1 accuracy action prediction accuracy
Baselines: VIOLET SigLIP CLIP fully supervised VPS/IPS models STTran TRACE GPA VT-TWINS TempCLR Supervised
Builds on: Scallop LTLf GPA TempCLR VT-TWINS
Closest papers by meaning
- Weakly Supervised Video Scene Graph Generation via Natural Language Supervision (ICLR 2025 · similarity 0.89)
- LLM4SGG: Large Language Models for Weakly Supervised Scene Graph Generation (CVPR 2024 · similarity 0.88)
- Towards Unbiased and Robust Spatio-Temporal Scene Graph Generation and Anticipation (CVPR 2025 · similarity 0.88)
- STPro: Spatial and Temporal Progressive Learning for Weakly Supervised Spatio-Temporal Grounding (CVPR 2025 · similarity 0.87)
- A Closer Look at Dynamic Scene Graph Generation in the Era of Multimodal Large Language Models (NeurIPS 2026 · similarity 0.87)
- Fine-Grained Captioning of Long Videos through Scene Graph Consolidation (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.