Research map / Video / Video-language understanding
TC-LLaVA: Rethinking the Transfer of LLava from Image to Video Understanding with Temporal Considerations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Adapts image-pretrained LLaVA-Next to video by modifying LLM attention with Temporal-Aware Dual RoPE and a Frame-wise Block Causal Attention Mask, achieving SOTA on several video QA and MVBench benchmarks with SFT only.
Problem
Image-pretrained MLLMs transferred to video understanding treat video frames as concatenated images, ignoring temporal position information and the distinct interaction pattern of visual tokens within and across frames inside the LLM's attention.
Approach
Improve the LLM's inter-layer attention for video by adding a temporal RoPE alongside the original RoPE (same position id for tokens in a frame, incrementing across frames) and by using a mask that is fully bidirectional within each frame but causal across frames.
Key results
- TC-LLaVA reports 56.6% average on MVBench, versus 54.3% for the LLaVA-Next-based baseline with original RoPE and causal mask.
- TC-LLaVA reaches 78.2% accuracy on TGIF-QA and 78.8% on MSVD-QA, the best reported among the compared models.
- On VCGBench, TC-LLaVA scores 3.19 average, slightly above the next-best reported value of 3.17 (GPT-4V).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add TC-Attention (ablation) | Vicuna-7B-v1.5 · MVBench | score: 54.3 -> 56.6 (+2.3) | Helped |
| add TC-Attention (ablation) | Vicuna-7B-v1.5 · VCGbench | score: 3.09 -> 3.19 (+0.1) | Helped |
| replace causal mask with Frame-wise Block Causal Mask (ablation) | TC-LLaVA with baseline RoPE · MVBench | score: 54.3 -> 55.9 | Helped |
| replace causal mask with Full Visual Mask (ablation) | TC-LLaVA with baseline RoPE · MVBench | score: 54.3 -> 53.8 | Hurt |
| replace causal mask with Frame-wise Block Mask (ablation) | TC-LLaVA with baseline RoPE · MVBench | score: 54.3 -> 54.0 | Hurt |
| add Time-Aware Dual RoPE while retaining global RoPE (ablation) | TC-LLaVA with baseline causal mask · MVBench | score: 54.3 -> 56.0 | Helped |
| replace global RoPE with Time-Aware RoPE alone (ablation) | TC-LLaVA with baseline causal mask · MVBench | score: 54.3 -> 54.2 | No clear effect |
| replace temporal RoPE with Time APE alongside original RoPE (ablation) | TC-LLaVA · MVBench | score: 54.3 -> 50.2 | Hurt |
| replace temporal RoPE with Time RPE alongside original RoPE (ablation) | TC-LLaVA · MVBench | score: 54.3 -> 52.7 | Hurt |
| add Time-Aware Dual RoPE (ablation) | TC-LLaVA · MVBench | score: 54.3 -> 56.0 | Helped |
| increase the Time-Aware RoPE ratio beyond 1.0 (ablation) | TC-LLaVA · MVBench | score | Hurt |
| add TC-Attention (ablation) | Llama3-8B-Instruct · MVBench | score: 56.2 -> 57.5 (+1.0) | Helped |
Surprising findings
- Time absolute and relative position encodings failed to converge during SFT, while Time RoPE trained stably.
- Full Visual Mask (bidirectional across all visual tokens) slightly underperformed the baseline on MVBench (53.8 vs 54.3).
Limitations
- Gains on several benchmarks are small (e.g., +0.1 VCGbench over GPT-4V-level numbers) and rely on a GPT-3.5 judge.
- Scaling factor gamma must be tuned manually; large gamma degrades performance.
- Evaluated only with 16 uniformly sampled frames and 7B-scale LLMs in the main results.
Experimental setup
Datasets: VideoChat Video-ChatGPT Kinetics SthSthV2 Webvid YouCook2 TextVR NextQA CLEVRER TGIF Ego4D MSVD-QA
Benchmarks: MSVD-QA MSRVTT-QA ActivityNet-QA TGIF-QA VCGBench MVBench
Metrics: accuracy score (0-5) VCG score across CI/DO/CU/TU/CO
Baselines: Video-LLaMA LLaMA-Adapter Video-ChatGPT Chat-UniVi MovieChat VideoChat VideoChat2 Vista-LLaMA LLaMA-VID IG-VLM LLaVA ST-LLM PLLaVA
Builds on: LLaVA-Next RoPE (RoFormer) Video-ChatGPT VideoChat2 PLLaVA
Closest papers by meaning
- Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders (NeurIPS 2025 · similarity 0.91)
- Understanding Long Videos with Multimodal Language Models (ICLR 2025 · similarity 0.90)
- LLaVA-ST: A Multimodal Large Language Model for Fine-Grained Spatial-Temporal Understanding (CVPR 2025 · similarity 0.90)
- TG-LLaVA: Text Guided LLaVA via Learnable Latent Embeddings (AAAI 2025 · similarity 0.90)
- LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token (ICLR 2025 · similarity 0.90)
- MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding (CVPR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.