Research map / Video / Video-language understanding
Unleashing Hour-Scale Video Training for Long Video-Language Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VideoMarathon, a 9.7K-hour synthetic hour-long video instruction dataset (3.3M QA over 22 tasks), and Hour-LLaVA, a Video-LMM that caches full 1-FPS video tokens in a memory repository and uses a MemAug cross-attention module to recover question-relevant detail from compressed tokens.
Problem
Hour-long video-language training is underexplored because well-annotated long-video instruction data is scarce, and existing Video-LMMs either sparsely sample frames (losing information) or process dense frames at prohibitive token cost.
Approach
Store full 1-FPS video tokens in a memory repository, feed only a compressed (~1/16 overall) set of decayed tokens to the LLM, and use learnable MemAug transformer blocks to re-inject question-relevant spatiotemporal details from the full cache, combined with a hierarchical (clip/event/global) synthetic captioning and QA pipeline for long videos.
Key results
- Hour-LLaVA-7B reaches 60.4 on LongVideoBench and 45.6 on LVBench, outperforming LLaVA-Video-7B (58.2 and 42.2).
- Hour-LLaVA-3B reaches 57.8 on LongVideoBench and 44.7 on LVBench, outperforming LLaVA-Video-3B (55.2 and 41.7).
- MemAug gives 1.4-2.4 point gains over the same token-compression baselines without it on LongVideoBench and LVBench.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add Hour-LLaVA long-video training and memory augmentation | LLaVA-Video-3B · LongVideoBench | M-Avg: 55.2 -> 57.8 | Helped |
| Add Hour-LLaVA long-video training and memory augmentation | LLaVA-Video-7B · LVBench | Avg: 42.2 -> 45.6 | Helped |
| Use random spatial forgetting at 1/4 compression instead of no token compression (ablation) | LLaVA-OV-SI-3B · MMStar | score: 52.8 -> 51.9 | Hurt |
| Use random spatial forgetting at 1/4 compression instead of no token compression (ablation) | LLaVA-OV-SI-3B · Sci.QA | score: 84.7 -> 84.5 | Hurt |
| Use uniform spatial forgetting at 1/4 compression instead of no token compression (ablation) | LLaVA-OV-SI-3B · MMStar | score: 52.8 -> 51.5 | Hurt |
| Replace random temporal forgetting with uniform temporal forgetting (ablation) | Hour-LLaVA-3B · LongVideoBench | M-Avg: 53.2 -> 54.0 | Helped |
| Add MemAug to uniform temporal compression (ablation) | Hour-LLaVA-3B · LongVideoBench | M-Avg: +1.9 | Helped |
| Add MemAug to uniform temporal compression (ablation) | Hour-LLaVA-3B · LVBench | Avg: +2.3 | Helped |
| Add 2 MemAug blocks instead of 1 (ablation) | Hour-LLaVA-3B · TempCompass | M-Avg: 59.4 -> 59.8 | Helped |
| Remove 1D structured RoPE from MemAug (ablation) | Hour-LLaVA-3B · Video-MME (Temporal Reasoning) | score: 46.3 -> 35.6 (-10.7) | Hurt |
Surprising findings
- LLaVA-Video trained on VideoMarathon fails to benefit from more long-video data and its performance declines, while Hour-LLaVA keeps improving.
- Performance with MemAug is largely insensitive to the choice of temporal forgetting strategy (uniform, keyframe, question-guided, random).
Limitations
- Multiple-choice QA is the main evaluation format, which limits assessment of broader Video-LMM capabilities.
- Training on large-scale synthetic data introduces noise that the training pipeline does not explicitly handle.
- Model is limited to video and language modalities and ignores audio.
Experimental setup
Datasets: VideoMarathon LLaVA-OneVision single-image data LLaVA-Video-178K Panda-70M Ego4D ActivityNet YouCook2 MovieChat-1K
Benchmarks: TempCompass LongVideoBench Video-MME LVBench MLVU-Dev MMStar ScienceQA RealWorldQA
Metrics: accuracy M-Avg average score (GPT-4-Turbo rating 2-10) human hallucination score (1-10) yes/no bias (Pct. Diff, FP Ratio)
Baselines: LLaVA-Video-3B LLaVA-Video-7B LLaVA-OV-SI Qwen2.5-VL-3B Qwen2-VL-7B Apollo LongVU LongVILA Kangaroo Video-XL
Builds on: LLaVA-Video LLaVA-OneVision SigLIP Qwen2.5
Closest papers by meaning
- HourVideo: 1-Hour Video-Language Understanding (NeurIPS 2024 · similarity 0.93)
- LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.92)
- LongVILA: Scaling Long-Context Visual Language Models for Long Videos (ICLR 2025 · similarity 0.91)
- E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding (NeurIPS 2024 · similarity 0.91)
- Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization (ICML 2024 · similarity 0.90)
- MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding (NeurIPS 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.