Research map / Video / Video-language understanding

World Model on Million-Length Video And Language With Blockwise RingAttention

ICLR 2025 · Video › Video-language understanding · approach: Video LLMs with fine-grained understanding · 12 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Trains LWM, a 7B Llama-2-based autoregressive model with 1M-token context over long text, video and image tokens (VQGAN), using progressive context extension, Blockwise RingAttention, and model-generated QA data.

Problem

Extending sequence models to million-token contexts for long text documents and hour-long videos is limited by memory/compute cost and by scarcity of long-form data and long-context chat data.

Approach

Progressively extend context from 32K to 1M by scaling RoPE theta, train with Blockwise RingAttention on books and then on mixed text-image-video tokens from a VQGAN tokenizer, and build long-context chat data by having a short-context model generate QA pairs from book chunks.

Key results

• LWM-Text-1M achieves 0.67/0.84/0.69 on multi-needle (N=2,R=2 / N=4,R=1 / N=4,R=2) at 1M context, where Llama-3.1-8B-Instruct, Qwen2.5-7B-Instruct and Mistral-7B-Instruct-v0.3 score at most 0.32 at 1M.
• On Video-MME, LWM-1M scores 63.7 medium and 60.8 long, the best among 7B-class models in the paper's table, though below LLaVA-Video 72B and Gemini 1.5 Pro.
• On LOFT at 512K context, LWM gets 0.38 Quora, 0.37 NQ and 0.72 HotpotQA, above GPT-4o (128K) and Claude 3 Opus (200K) on all three.

Measured effects reported in the paper

Surprising findings

• Image understanding with discrete VQGAN tokens lags CLIP-based models considerably (e.g., VQAv2 55.8 vs LLaVA-1.5 78.5), which the authors attribute to information loss in discrete tokens.
• MT-Bench score falls and needle accuracy drops as the chat share of training data increases, showing a trade-off between chat ability and retrieval.

Limitations

• Image and video tokenization uses a vanilla VQGAN with frame-by-frame video tokens, causing information loss and weaker OCR-like and short-video performance.
• Model scale is limited to 7B, so findings may not transfer to 100B+ models.
• Multi-needle retrieval degrades with harder settings, indicating room to improve 1M context utilization.

Experimental setup

Datasets: Books3 (The Pile) UltraChat LAION-2B-en COYO-700M WebVid10M InternVid10M ShareGPT4V Valley-Instruct-73K Video-ChatGPT-100K OpenLLaMA data

Benchmarks: ARC-Challenge HellaSwag MMLU OpenBookQA Needle In A Haystack Multi-needle retrieval LOFT (Quora, NQ, HotpotQA) Video-MME VQAv2 GQA ScienceQA MSVD

Metrics: accuracy normalized accuracy retrieval accuracy MT-Bench score

Baselines: Gemini Pro GPT-4-1106 Gemini 1.5 Pro GPT-4o Claude 3 Opus LLaVA-Video Video-LLaVA LLaVA-1.5 Qwen2.5-7B-Instruct Llama-3.1-8B-Instruct Mistral-7B-Instruct-v0.3

Builds on: Blockwise RingAttention FlashAttention Llama 2 RoPE VQGAN

Closest papers by meaning

• LongVILA: Scaling Long-Context Visual Language Models for Long Videos (ICLR 2025 · similarity 0.91)
• Data Engineering for Scaling Language Models to 128K Context (ICML 2024 · similarity 0.89)
• Unleashing Hour-Scale Video Training for Long Video-Language Understanding (NeurIPS 2025 · similarity 0.89)
• MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding (CVPR 2024 · similarity 0.89)
• A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.88)
• LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding (NeurIPS 2024 · similarity 0.88)