Research map / Video / Video-language understanding
HIG: Hierarchical Interlacement Graph Approach to Scene Graph Generation in Video Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the ASPIRe video dataset with five interactivity attribute types and the Hierarchical Interlacement Graph (HIG) for scene graph generation over video hierarchies.
Problem
Visual interactivity understanding in videos requires capturing diverse appearance, situation, position, interaction, and relation attributes between humans and objects over time, which existing datasets and scene graph methods cover poorly.
Approach
Model a video as a hierarchy of graphs in which each level merges graphs from the previous level via message passing with shared per-level weights, so a top-level graph cell summarizes the whole video, and predict single-actor and double-actor attributes from the top level.
Key results
- On ASPIRe double-actor attributes at R@100, HIG reaches 42.33 (position), 41.65 (interaction), 41.47 (relation) versus Transformer-based baseline 38.84, 40.70, 39.90 (Table 4).
- On PSG, HIG achieves R@100 of 26.3 versus VCTree's 22.5, a 3.8-point gain at R@100 (Table 7).
- On the SGG comparison (Table 6), HIG improves over GPSNet by 3.55, 5.82, and 6.73 points at R@100 for position, interaction, and relation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use HIG rather than GPSNet | Video scene graph generation · ASPIRe, position | R@100: 12.89 -> 42.33 | Helped |
| use HIG rather than GPSNet | Video scene graph generation · ASPIRe, interaction | R@100: 10.89 -> 41.65 | Helped |
| use HIG rather than GPSNet | Video scene graph generation · ASPIRe, relation | R@100: 9.87 -> 41.47 | Helped |
| use HIG rather than GPSNet | Image scene graph generation · SGG, position | R@100: 1.32 -> 4.87 | Helped |
| use HIG rather than GPSNet | Image scene graph generation · SGG, interaction | R@100: 1.11 -> 6.93 | Helped |
| use HIG rather than GPSNet | Image scene graph generation · SGG, relation | R@100: 0.29 -> 7.02 | Helped |
| halve the number of video frames (ablation) | HIG · ASPIRe, position | R@20: 13.02 / 0.09 -> 10.13 / 0.08 | Hurt |
| halve the number of video frames (ablation) | HIG · ASPIRe | FPS: 24.2 -> 26.4 | Helped |
| apply HIG to images as single-frame videos | HIG · PSG | R@100: 22.5 -> 26.3 | Helped |
Surprising findings
- On PSG, HIG is slightly below PSGFormer and the other baselines at R@20, although it is higher at R@100.
Limitations
- Computing possible interlacements is a computational bottleneck that may hinder real-time use.
- Handling long-duration videos is difficult, and continual learning of new interactivities can cause forgetting.
- The model is tailored for video, so image-based performance may be suboptimal.
Experimental setup
Datasets: ASPIRe PSG
Benchmarks: ASPIRe PSG
Metrics: R@K mR@K
Baselines: IMP MOTIFS VCTree GPSNet PSGFormer Transformer-based baseline Convolution-based baseline Handcrafted baseline Vanilla baseline
Builds on: Action Genome PSG PVSG Focal Loss DETR
Closest papers by meaning
- HyperGLM: HyperGraph for Video Scene Graph Generation and Anticipation (CVPR 2025 · similarity 0.88)
- Adaptive Visual Scene Understanding: Incremental Scene Graph Generation (NeurIPS 2024 · similarity 0.87)
- Multiview Scene Graph (NeurIPS 2024 · similarity 0.86)
- A Closer Look at Dynamic Scene Graph Generation in the Era of Multimodal Large Language Models (NeurIPS 2026 · similarity 0.86)
- Scene Graph-Grounded Image Generation (AAAI 2025 · similarity 0.86)
- Learning to Evolve Scenes: Reasoning about Human Activities with Scene Graphs (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.