Research map / Video / Video-language understanding
Compositional Video Understanding with Spatiotemporal Structure-based Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ST-GT, a spatiotemporal graph transformer over object-centric symbolic graphs with a compositional disentangled-embedding loss, and new compositional generalization splits on CATER, reporting SOTA on CATER and MOMA-LRG.
Problem
Understanding semantic structure in long videos and generalizing compositionally to unseen combinations of objects, actions, and temporal relations.
Approach
Represents each video as a spatiotemporal symbolic graph tokenized into node, spatial-edge, and temporal-edge tokens for a graph transformer, then disentangles object and action embeddings with auxiliary losses so that unseen label combinations are predicted from seen semantic elements.
Key results
- Task 1 on CATER: 99.88 mAP vs 98.9 for R3D + NL.
- Task 2 on CATER: 75.40 mAP vs 66.71 for SCI3D + LSTM and 66.18 for ViViT.
- MOMA-LRG: 72.83 mAP vs 69.36 for FROZEN, with 7.3M parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add compositional loss Lcomp to Lvid (ablation) | Ours · CATER Task 1 validation | mAP: 90.42 -> 99.89 | Helped |
| add compositional loss Lcomp to Lvid (ablation) | Ours · CATER Task 1 test | mAP: 90.38 -> 99.88 | Helped |
| add compositional loss Lcomp to Lvid (ablation) | Ours · CATER Task 2 validation | mAP: 56.16 -> 76.07 | Helped |
| add compositional loss Lcomp to Lvid (ablation) | Ours · CATER Task 2 test | mAP: 53.53 -> 75.40 | Helped |
| include node, spatial-edge, and temporal-edge tokens in ST-GT (ablation) | Ours · CATER Task 2 | mAP: 72.49 -> 75.40 | Helped |
| include node, spatial-edge, and temporal-edge tokens in ST-GT (ablation) | Ours · MOMA-LRG | mAP: 49.76 -> 72.83 | Helped |
| use the proposed method instead of the strongest listed baseline | CATER Task 1 · CATER Task 1 | mAP: 98.9 -> 99.88 | Helped |
| use the proposed method instead of the strongest listed baseline | CATER Task 2 · CATER Task 2 | mAP: 69.76 -> 75.40 | Helped |
| use the proposed method instead of the strongest listed baseline | MOMA-LRG · MOMA-LRG | mAP: 69.36 -> 72.83 | Helped |
| use the proposed method instead of the R3D + LSTM baseline | CATER Task 2 compositional generalization · CATER Task 2 D2 test test | performance: 3.51 -> 69.2 | Helped |
Surprising findings
- On the compositional generalization split for Task 2, R3D baselines drop to about 2-4% accuracy while the proposed model reaches over 80%.
Limitations
- Symbolic graphs are assumed given or built via object detection and attribute classification, which the paper says is easy only for CATER videos.
- Compositional generalization evaluation is limited to CATER splits constructed by the authors.
Experimental setup
Datasets: CATER MOMA-LRG
Benchmarks: CATER MOMA-LRG
Metrics: mAP mean recall accuracy
Baselines: FasterRCNN SCI3D R3D R3D + NL R3D + LSTM ViViT FROZEN
Builds on: TokenGT GPT-2 I3D SCI3D
Closest papers by meaning
- Understanding Video Transformers via Universal Concept Discovery (CVPR 2024 · similarity 0.86)
- STEP: Enhancing Video-LLMs' Compositional Reasoning by Spatio-Temporal Graph-guided Self-Training (CVPR 2025 · similarity 0.86)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.85)
- Re-thinking Temporal Search for Long-Form Video Understanding (CVPR 2025 · similarity 0.85)
- VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLM (CVPR 2025 · similarity 0.85)
- Streaming Long Video Understanding with Large Language Models (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.