Research map / Vision-language and multimodal / Visual question answering
Cross-modal Causal Relation Alignment for Video Question Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CRA, a weakly supervised video question grounding framework that uses Gaussian-smoothed cross-modal attention for temporal grounding, bidirectional contrastive alignment, and front-door/back-door causal interventions, trained only on VideoQA supervision.
Problem
Video question grounding (VideoQG) must answer questions and localize supporting video segments, but existing models rely on spurious cross-modal correlations and unfaithful grounding, and annotated grounding data is scarce.
Approach
Uses VideoQA supervision as weak signal to learn cross-modal attention-based grounding with adaptive Gaussian smoothing, aligns video and QA features via bidirectional InfoNCE, and applies front-door intervention on visual mediators plus back-door intervention on linguistic confounders from semantic structure graphs.
Key results
- On NextGQA test, CRA with Temp[CLIP] reaches Acc@GQA 18.2 and IoU@0.5 10.6, matching TimeCraft's Acc@GQA and beating it on IoU@0.5 (9.6).
- With FrozenBiLM backbone, CRA achieves Acc@GQA 18.8 vs TimeCraft 18.5.
- On STAR val, CRA with Temp[CLIP] achieves Acc@GQA 26.8 vs 24.4 for NG+ baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add GSG to CRA (ablation) | CRA on NextGQA · NextGQA test | Acc@GQA: 16.4 (↓1.8) -> 18.2 | Helped |
| add GSG to CRA (ablation) | CRA on NextGQA · NextGQA test | TIoU@0.5: 8.0 (↓2.6) -> 10.6 | Helped |
| add CMA bidirectional contrastive alignment (ablation) | CRA on NextGQA · NextGQA test | Acc@GQA: 16.2 (↓2.0) -> 18.2 | Helped |
| add CMA bidirectional contrastive alignment (ablation) | CRA on NextGQA · NextGQA test | TIoP@0.5: 26.3 (↓2.2) -> 28.5 | Helped |
| add causal intervention modules (ablation) | CRA on NextGQA · NextGQA test | Acc@GQA: 17.0 (↓1.2) -> 18.2 | Helped |
| add LCI language deconfounding (ablation) | CRA on NextGQA · NextGQA test | Acc@GQA: 17.8 (↓0.4) -> 18.2 | Helped |
| add ECI visual causal intervention (ablation) | CRA on NextGQA · NextGQA test | TIoP@0.5: 27.3 (↓1.2) -> 28.5 | Helped |
| add Gaussian smoothing during training (ablation) | CRA GSG on NextGQA · NextGQA test | TIoU@0.5: 8.4 -> 10.6 | Helped |
| disable Gaussian smoothing at inference (ablation) | CRA GSG on NextGQA · NextGQA test | Acc@GQA: 18.2 -> 15.8 | Hurt |
| use CRA instead of NG+ interval generation | Temp[CLIP] on STAR · STAR val | Acc@GQA: 24.4 -> 26.8 | Helped |
| use CRA instead of NG+ interval generation | FrozenBiLM on STAR · STAR val | TIoU@0.5: 7.8 -> 5.1 | Hurt |
Surprising findings
- Large VLM backbones like FrozenBiLM and SeViLA achieve high Acc@VQA but very low Acc@GQA, indicating unfaithful answers despite correct predictions.
Limitations
- Generated intervals concentrate around about 2.5s and underestimate long segments.
- Grounding is weakly supervised and relies on VideoQA signal, with lower IoU at strict thresholds.
Experimental setup
Datasets: NextGQA STAR
Benchmarks: NextGQA STAR
Metrics: Acc@GQA Acc@VQA mIoP mIoU IoP@0.3 IoP@0.5 IoU@0.3 IoU@0.5
Baselines: IGV SeViLA VIOLETv2 VGT Temp[CLIP] Temp[Swin] Temp[BLIP] FrozenBiLM TimeCraft
Builds on: Temp[CLIP] IGV FrozenBiLM NWGM
Closest papers by meaning
- Multi-Pair Temporal Sentence Grounding via Multi-Thread Knowledge Transfer Network (AAAI 2025 · similarity 0.87)
- Searching Videos as Trees: Self-Correcting Agents for Grounded Long Video QA (NeurIPS 2026 · similarity 0.86)
- Align and Aggregate: Compositional Reasoning with Video Alignment and Answer Aggregation for Video Question-Answering (CVPR 2024 · similarity 0.86)
- Two Causally Related Needles in a Video Haystack (NeurIPS 2025 · similarity 0.86)
- Video-Context Aligned Transformer for Video Question Answering (AAAI 2024 · similarity 0.86)
- Can I Trust Your Answer? Visually Grounded Video Question Answering (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.