Research map / Video / Video-language understanding
STAIR: Spatial-Temporal Reasoning with Auditable Intermediate Results for Video Question Answering
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STAIR, a neural module network for video QA with 16 lightweight modules that return auditable intermediate outputs, trained with intermediate supervision from scene-graph programs.
Problem
Video QA models struggle with temporal reasoning on long, informative videos, and existing neural module networks for video are underdeveloped and their intermediate outputs are often attention maps that are hard to inspect.
Approach
Define video-text sub-tasks as neural modules that return intention-specific intermediate outputs (e.g., object vectors, frame attention), generate programs with a seq2seq LSTM generator, and supervise intermediate modules with ground-truth outputs derived from AGQA scene-graph programs.
Key results
- On AGQA with I3D features, STAIR-IS reaches 55.06 overall accuracy vs 52.28 for MAC (I3D) and 50.75 for STAIR with RX features.
- On AGQA2, STAIR (I3D) reaches 52.06 overall vs 49.67 for MAC.
- Adding STAIR-IS Filter outputs as prompts raises Video-ChatGPT overall accuracy on AGQA2 from 35.09 to 40.43 (GPT-3.5 judged), and GPT-2+STAIR-IS reaches 57.34 on AGQA vs 57.14 for GPT-2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add intermediate supervision (ablation) | STAIR with RX features · AGQA | Overall accuracy: 50.75 -> 51.14 | Helped |
| add intermediate supervision (ablation) | STAIR with I3D features · AGQA | Overall accuracy: 53.45 -> 55.06 | Helped |
| add intermediate supervision (ablation) | STAIR with I3D features · AGQA2 | Overall accuracy: 52.06 -> 51.41 | Hurt |
| add intermediate supervision (ablation) | STAIR with I3D features · AGQA | Filter R@5: 0.30 -> 0.50 | Helped |
| add intermediate supervision (ablation) | STAIR with I3D features · AGQA | Localize IoU: 0.19 -> 0.23 | Helped |
| use I3D instead of RX video features (ablation) | STAIR · AGQA | Overall accuracy: 50.75 -> 53.45 | Helped |
| use STAIR-IS intermediate results as prompts | GPT-2 · AGQA | Overall accuracy: 57.14 -> 57.34 | Helped |
| use STAIR-IS intermediate results as prompts | GPT-2 · AGQA2 | Overall accuracy: 52.96 -> 55.13 | Helped |
| add STAIR-IS intermediate results | Video-ChatGPT · AGQA2 (sampled 1%) | Accuracy: 35.09 (0.76) -> 40.43 (0.89) | Helped |
Surprising findings
- Intermediate supervision does not consistently improve final accuracy (STAIR-IS is worse than STAIR on AGQA2), though it improves intermediate output accuracy.
- STAIR underperforms pre-trained GPT-2 and Violet on AGQA, while the gap narrows on AGQA2.
Limitations
- Intermediate supervision requires scene-graph-derived ground truth, which is only available for AGQA-style data; about 5% of training examples lack correct ground truth.
- Performs worse than pre-trained models such as ClipBERT on MSRVTT-QA and underperforms on Interaction-type STAR questions.
- Relies on program annotations for training the program generator on AGQA2 for transfer experiments.
Experimental setup
Datasets: AGQA AGQA2 STAR MSRVTT-QA Kinetics Charades Action Genome
Benchmarks: AGQA AGQA2 STAR MSRVTT-QA
Metrics: accuracy Recall@N IoU_att
Baselines: PSAC HME HCRN MAC DSTN-E2E GPT-2 Violet Video-ChatGPT CNN-BERT L-GCN ClipBERT Co-Memory
Builds on: Neural Module Networks HCRN MAC DSTN
Closest papers by meaning
- Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Task (NeurIPS 2025 · similarity 0.89)
- Enhancing Video-LLM Reasoning via Agent-of-Thoughts Distillation (CVPR 2025 · similarity 0.89)
- MoReVQA: Exploring Modular Reasoning Models for Video Question Answering (CVPR 2024 · similarity 0.88)
- Language-aware Visual Semantic Distillation for Video Question Answering (CVPR 2024 · similarity 0.88)
- Divide and Conquer: Exploring Language-centric Tree Reasoning for Video Question-Answering (ICML 2025 · similarity 0.88)
- VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.