Research map / Video / Video-language understanding
HyperGLM: HyperGraph for Video Scene Graph Generation and Anticipation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HyperGLM, which injects a unified scene hypergraph (entity scene graphs plus a procedural transition graph) into a multimodal LLM for video scene graph generation, anticipation and reasoning, and releases the 1.9M-frame VSGR dataset.
Problem
Video scene graph generation and anticipation methods rely on pairwise object relations and struggle with multi-way interactions, higher-order relations, and reasoning tasks such as VQA and captioning.
Approach
Builds a unified hypergraph by merging per-frame entity scene graphs with a relationship-transition procedural graph, samples hyperedges via random walks, and injects this structure into an MLLM (CLIP encoder, MLP projector, Mistral-7B with LoRA) for reasoning over video.
Key results
- SGA on Action Genome: HyperGLM reaches R@10 of 38.8% (vs 37.3% for SceneSayerSDE) at the 0.9 input fraction reported in Table 4.
- SGG on VSGR: HyperGLM achieves R@20 of 35.8% and mR@100 of 10.4%, above the HyperGraph variant and prior baselines.
- VQA on VSGR: HyperGLM reaches 45.4% accuracy, above Chat-UniVi-7B at 44.3%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HyperGLM instead of CYCLO | Video scene graph generation · VSGR | R@20: 29.4 -> 35.8 | Helped |
| Use HyperGLM instead of CYCLO | Video scene graph generation · PVSG | mR@20: 2.0 -> 2.8 | Helped |
| Use HyperGLM instead of SceneSayerSDE | Scene graph anticipation · Action Genome | R@10: 37.3 -> 38.8 | Helped |
| Use HyperGLM instead of SceneSayerSDE | Scene graph anticipation · VSGR | R@10: 27.5 -> 30.2 | Helped |
| Use HyperGLM instead of Chat-UniVi-7B | Video question answering · VSGR | accuracy: 44.3 -> 45.4 | Helped |
| Use HyperGLM instead of MV-GPT | Video captioning · VSGR | CIDEr: 57.1 -> 54.5 | Hurt |
| Use HyperGLM instead of MV-GPT | Video captioning · VSGR | ROUGE-L: 62.5 -> 64.9 | Helped |
| Use HyperGLM instead of LLaMA-VID-7B | Relation reasoning · VSGR | accuracy: 44.1 -> 47.2 | Helped |
| Inject the unified HyperGraph into the LLM (ablation) | HyperGraph · VSGR | R@20: 31.6 -> 35.8 | Helped |
| Increase the observed video fraction from 0.3 to 0.7 (ablation) | HyperGLM scene graph anticipation · VSGR | R@10: 19.0 -> 25.1 | Helped |
| Increase the observed video fraction from 0.3 to 0.7 (ablation) | HyperGLM scene graph anticipation · Action Genome | R@10: 27.5 -> 35.7 | Helped |
| Use 60 hyperedges (Nw=60, Nl=7) (ablation) | HyperGLM scene graph generation · VSGR | Recall | Helped |
Surprising findings
- The HyperGraph-only variant without the LLM scores lower than the full HyperGLM, and in some SGG cells is also below the pairwise baseline on R@20 of VSGR.
Limitations
- Managing many objects and their interactions can complicate the hypergraph and obscure essential relationships.
- Relies on a pre-trained object detector (Faster R-CNN) for SGG and SGA settings.
Experimental setup
Datasets: VSGR PVSG Action Genome ASPIRe AeroEye MSRVTT-QA MSVD-QA
Benchmarks: PVSG Action Genome VSGR
Metrics: Recall@K mean Recall@K Accuracy Precision Recall F1 CIDEr METEOR ROUGE-L BLEU-4
Baselines: STTran+ DSGDetr+ STTran++ DSGDetr++ SceneSayerODE SceneSayerSDE Transformer HIG CYCLO Video-ChatGPT Video-LLaVA-7B MovieChat
Builds on: SceneSayer STTran DSGDetr LLaVA
Closest papers by meaning
- A Closer Look at Dynamic Scene Graph Generation in the Era of Multimodal Large Language Models (NeurIPS 2026 · similarity 0.89)
- Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models (ICML 2026 · similarity 0.89)
- HIG: Hierarchical Interlacement Graph Approach to Scene Graph Generation in Video Understanding (CVPR 2024 · similarity 0.88)
- From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models (CVPR 2024 · similarity 0.88)
- DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation (CVPR 2024 · similarity 0.88)
- Synthetic Visual Genome (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.