Research map / Video / Video-language understanding
Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adds stacked temporal attention blocks inside the vision encoder of Qwen2-VL (STAVEQ2) and related Video-LLMs, trained in two stages on WebVid-QA, to improve temporal understanding on SSv2-T10, VITATECS, MVBench and Video-MME.
Problem
Video-LLMs struggle to distinguish temporally mirrored actions (e.g., pulling left-to-right vs right-to-left) and fail at in-context learning for video recognition, which the authors attribute to missing temporal modeling in the vision encoder.
Approach
Insert temporal self-attention blocks after spatial attention in each vision-encoder transformer block (with fewer heads and 1D RoPE), initialized with zero output projections and trained first alone then jointly with LoRA adapters on the LLM, so the visual tokens carry inter-frame dynamics before reaching the LLM.
Key results
- InternVideo2 1B + STA reaches 78.0% on full SSv2, above InternVideo2 6B (77.5%).
- STAVEQ2 7B reaches 70.1 on MVBench vs 67.0 for Qwen2-VL 7B, and 66.8/71.8 on Video-MME (wo/w subtitles) vs 63.3/69.0.
- On SSv2-T10, InternVideo2-Chat 8B improves from 84.17% to 95.18% with STA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add stacked temporal attention | InternVideo2 1B · SSv2 | accuracy: 77.1 -> 78.0 | Helped |
| Add stacked temporal attention | InternVideo2-Chat 8B · SSv2-T10 | accuracy: 84.17% -> 95.18%; 11.01% | Helped |
| Add stacked temporal attention | Qwen2-VL 2B · SSv2-VSM | accuracy: 68.65 -> 72.19 | Helped |
| Add stacked temporal attention | Qwen2-VL 7B · SSv2-VSM | accuracy: 73.15 -> 76.05 | Helped |
| Add stacked temporal attention | Qwen2-VL 7B · MVBench | accuracy: 67.0 -> 70.1 | Helped |
| Use spatial attention before temporal attention with head scale 0.25 (ablation) | STAVEQ2 2B · SSv2-T10 | accuracy: 73.14 -> 76.04 | Helped |
| Place temporal attention before spatial attention instead of after it (ablation) | STAVEQ2 2B, head scale 0.25 · SSv2-T10 | accuracy: 76.04 -> 73.20 | Hurt |
| Reduce temporal head scale from 1.0 to 0.5 (ablation) | STAVEQ2 2B, spatial first · SSv2-T10 | accuracy: 58.34 -> 71.18 | Helped |
| Use 32 temporal blocks across all layers instead of 16 (ablation) | STAVEQ2 2B · SSv2-T10 | accuracy: 74.73 -> 76.04; 74.97 -> 76.04 | Helped |
| Increase positive training samples from 50% to 80% (ablation) | SSv2-VSM context-selection fine-tuning · SSv2-VSM similarity matching | accuracy: 25.52% -> 71.25% | Helped |
| Increase positive training samples from 80% to 91% (ablation) | SSv2-VSM context-selection fine-tuning · SSv2-VSM similarity matching | accuracy: 71.25% -> 49.18% | Hurt |
| Remove textual descriptions of reference actions (ablation) | SSv2-VSM context-selection fine-tuning · SSv2-VSM similarity matching | accuracy: 71.25% -> 68.65% | Hurt |
Surprising findings
- Adding in-context examples generally hurt accuracy on SSv2-T10 for most Video-LLMs, and InternVideo2-Chat's accuracy fell from 30.60% zero-shot to 10.41% with 5 examples.
- InternVideo2-Chat is highly sensitive to intra-demonstration modality order (37.06% with video-first vs 15.98% with text-first on SSv2-VSM).
Limitations
- No pretraining or training from scratch due to resource constraints.
- Experiments limited to models up to 72B parameters.
- Several baseline numbers are missing ('--') for VITATECS and MVBench, and Qwen2-VL/Qwen2.5-VL VITATECS results were produced by the authors.
Experimental setup
Datasets: Something-Something v2 SSv2-T10 SSv2-VSM WebVid-QA WebVid
Benchmarks: SSv2-T10 SSv2 (full) SSv2-VSM VITATECS MVBench Video-MME
Metrics: accuracy
Baselines: Qwen2-VL Qwen2.5-VL InternVideo2-Chat LLaVA-NeXT-Video ST-LLM TG-Vid LLaVA-OneVision VideoLLaMA2 LLaVA-Video GPT-4o VideoMAE V2 MVD
Builds on: Qwen2-VL InternVideo2 VideoRoPE LoRA Vision Transformer
Closest papers by meaning
- TC-LLaVA: Rethinking the Transfer of LLava from Image to Video Understanding with Temporal Considerations (AAAI 2025 · similarity 0.91)
- TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding (CVPR 2024 · similarity 0.91)
- MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding (CVPR 2024 · similarity 0.91)
- SlowFocus: Enhancing Fine-grained Temporal Understanding in Video LLM (NeurIPS 2024 · similarity 0.91)
- Temporal Reasoning Transfer from Text to Video (ICLR 2025 · similarity 0.90)
- Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoning (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.