Research map / Vision-language and multimodal / Cross-modal learning
Dissecting Multimodality in VideoQA Transformer Models by Impairing Modality Fusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QUAG, a non-parametric inference-time probe that block-averages attention quadrants to impair modality fusion in VideoQA Transformers, plus QUAG-attention and the CLAVI stress-test dataset, showing that many models succeed without multimodal fusion and fail on highly coupled temporal questions.
Problem
Unclear whether VideoQA Transformers actually learn to jointly leverage video and text modalities or instead exploit dataset biases and unimodal shortcuts to reach high benchmark accuracy.
Approach
QUAG block-averages quadrants of the self-attention matrix (video-video, video-text, text-video, text-text) at inference time to ablate specific modality interactions without finetuning; QUAG-attention turns this into a cheaper restricted attention by pruning averaged keys/values; CLAVI builds temporal complement videos and questions with high modality coupling.
Key results
- FrozenBiLM drops to near chance (about 20%) on NeXT-QA and ATP-Hard under unimodal and text short-circuiting, while JustAsk drops by less than 1% on all datasets under any short-circuit operation.
- QUAG-attention reduces multiplication operations by 13.6% for FrozenBiLM and 68.0% for JustAsk with less than 1% accuracy drop.
- On CLAVI complement subset, JustAsk, Singularity-T and All-In-One+ achieve below 4% video- and text-consistent accuracy, while FrozenBiLM reaches 54.1% and 57.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace self-attention with QUAG-attention | FrozenBiLM and JustAsk · ActivityNet-QA, MSRVTT-QA, NeXT-QA, ATP-Hard | accuracy: FrozenBiLM: 43.6 -> 43.0, 46.6 -> 45.8, 55.8 -> 55.6, 55.7 -> 55.9; JustAsk: 38.7 -> 38.0, 41.8 -> 41.0, 53.8 -> 53.5, 4 | No clear effect |
| Use QUAG-attention to reduce multiplication operations | FrozenBiLM · ActivityNet-QA, MSRVTT-QA, NeXT-QA, ATP-Hard | multiplication operations: 13.6% | Helped |
| Use QUAG-attention to reduce multiplication operations | JustAsk · ActivityNet-QA, MSRVTT-QA, NeXT-QA, ATP-Hard | multiplication operations: 68.0% | Helped |
| Short-circuit unimodal attention quadrants (ablation) | FrozenBiLM · ActivityNet-QA and MSRVTT-QA | accuracy: 43.6 -> 2.4; 46.6 -> 1.0 | Hurt |
| Short-circuit crossmodal attention quadrants (ablation) | FrozenBiLM · ActivityNet-QA and MSRVTT-QA | accuracy: 43.6 -> 32.3; 46.6 -> 32.8 | Hurt |
| Short-circuit video attention quadrants (ablation) | FrozenBiLM · ActivityNet-QA and MSRVTT-QA | accuracy: 43.6 -> 43.1; 46.6 -> 45.7 | No clear effect |
| Short-circuit text attention quadrants (ablation) | FrozenBiLM · NeXT-QA and ATP-Hard | accuracy: 55.8 -> 20.5; 55.7 -> 21.1 | Hurt |
| Short-circuit unimodal, crossmodal, video, or text attention quadrants (ablation) | JustAsk · ActivityNet-QA, MSRVTT-QA, NeXT-QA, ATP-Hard | accuracy: ActivityNet-QA: 38.7 -> 38.5, 38.3, 38.2, 38.6; MSRVTT-QA: 41.8 -> 41.5, 41.3, 41.3, 41.5; NeXT-QA: 53.8 -> 53.6, 53.5, | No clear effect |
| Finetune VideoQA models on CLAVI's high-coupling complement subset | JustAsk, Singularity-T, and All-In-One+ · CLAVI complement subset | CAcc_V: 3.6~\textpm~0.1; 1.7~\textpm~0.2; 1.2~\textpm~0.3 | Hurt |
| Apply video-average QUAG-attention | FrozenBiLM · CLAVI complement subset | CAcc_T: 57.2% -> 23% | Hurt |
| Apply video-average QUAG-attention | FrozenBiLM · CLAVI complement subset | CAcc_V: 54.1~\textpm~0.2 -> 3.6% | Hurt |
Surprising findings
- JustAsk's accuracy is essentially unchanged under all short-circuit operations, suggesting it does not use cross-modal fusion on these datasets.
- Most models with over 70% balanced accuracy on CLAVI score near-trivially on the temporal complement subset.
Limitations
- CLAVI is limited to simple yes/no temporal questions built from Charades-STA templates and is not meant to replace existing benchmarks.
- Analysis covers only a few models (JustAsk, FrozenBiLM) on a fixed set of datasets; coupling is not directly measurable on real data.
Experimental setup
Datasets: ActivityNet-QA MSRVTT-QA NeXT-QA ATP-Hard Charades-STA HowToVQA69M WebVidVQA3M WebVid10M
Benchmarks: ActivityNet-QA MSRVTT-QA NeXT-QA ATP-Hard CLAVI
Metrics: accuracy balanced accuracy consistent accuracy (CAcc) percentage change in multiplication operations
Baselines: JustAsk FrozenBiLM Singularity-Temporal All-In-One+
Builds on: self-attention transformer FrozenBiLM JustAsk Charades-STA
Closest papers by meaning
- CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusion (ICLR 2025 · similarity 0.88)
- Apollo: An Exploration of Video Understanding in Large Multimodal Models (CVPR 2025 · similarity 0.88)
- Fusion or Confusion? Multimodal Complexity Is Not All You Need (NeurIPS 2026 · similarity 0.88)
- Mirasol3B: A Multimodal Autoregressive Model for Time-Aligned and Contextual Modalities (CVPR 2024 · similarity 0.87)
- MVBench: A Comprehensive Multi-modal Video Understanding Benchmark (CVPR 2024 · similarity 0.87)
- InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.