Research map / Video / Video-language understanding
Learning Situation Hyper-Graphs for Video Question Answering
Aisha Urooj, Hilde Kuehne, Bo Wu, Kim Chheu, Walid Bousselham, Chuang Gan, Niels Lobo, Mubarak Shah
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Predict action and relationship situation hyper-graphs and use them for question reasoning | Video question answering · AGQA 2.0 Balanced test | accuracy (all): 42.11 vs. 49.20 | Improved |
| Predict situation hyper-graphs and reason over them for video QA | Video question answering · STAR test | accuracy (Overall): 36.70 -> 39.47 | Improved |
| Use ground-truth rather than predicted hyper-graphs (ablation) | SHG-VQA · STAR validation | accuracy (Overall): 40.26 -> 62.46 | Improved |
| Add video embeddings to question and hyper-graph cross-attention inputs (ablation) | SHG-VQA · STAR test | accuracy (Overall): 39.47 -> 39.18 | No clear effect |
| Use both action and relationship tokens rather than action tokens only (ablation) | SHG-VQA · STAR validation (single model) | accuracy (Overall): 38.68 -> 39.20 | Improved |
| Use both action and relationship tokens rather than relationship tokens only (ablation) | SHG-VQA · STAR validation (single model) | accuracy (Overall): 35.16 -> 39.20 | Improved |
| Increase relation queries from 8 to 12 (ablation) | SHG-VQA · STAR validation (single model) | accuracy (Overall): 39.20 -> 39.90 | Improved |
| Increase action queries from 3 to 4 with 8 relation queries (ablation) | SHG-VQA · STAR validation (single model) | accuracy (Overall): 39.20 -> 37.06 | Hurt |
| Reduce action queries from 3 to 2 with 8 relation queries (ablation) | SHG-VQA · STAR validation (single model) | accuracy (Overall): 39.20 -> 38.34 | Hurt |
| Perform set matching frame by frame rather than over the full video (ablation) | SHG-VQA · STAR test | accuracy (Overall): 35.15 -> 35.63 | Mixed |
| Train with 100K rather than 1.4M QA pairs (ablation) | SHG-VQA · AGQA test | accuracy (all): 49.20 -> 43.69 | Hurt |
| Remove visual and hyper-graph inputs, retaining question-only information (ablation) | SHG-VQA · AGQA test | accuracy (all): 49.20 -> 40.95 | Hurt |
Closest papers by meaning
- Location-Aware Graph Convolutional Networks for Video Question Answering (AAAI 2020 · similarity 0.90)
- Video Graph Transformer for Video Question Answering (ECCV 2022 · similarity 0.88)
- Relation-Aware Graph Attention Network for Visual Question Answering (ICCV 2019 · similarity 0.88)
- (2.5+1)D Spatio-Temporal Scene Graphs for Video Question Answering (AAAI 2022 · similarity 0.88)
- Graph-Structured Representations for Visual Question Answering (CVPR 2017 · similarity 0.87)
- G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
