Research map / Video / Video-language understanding
Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Video-LaVIT decomposes videos into keyframes (LaVIT image tokens) and motion vectors (a learned VQ motion tokenizer), enabling unified next-token generative pre-training of Llama 2 7B over video, image, and text, with an EDM-based video detokenizer for generation.
Problem
Video-language pre-training with LLMs is costly because per-frame or 3D video tokenizers produce long token sequences, and existing LLM-based approaches struggle to model temporal dynamics efficiently for both understanding and generation.
Approach
Represent each video clip as one keyframe plus motion vectors (from MPEG-4 decoding), tokenize the keyframe with the LaVIT image tokenizer and the motion vectors with a spatiotemporal VQ-VAE, then train an LLM autoregressively on alternating visual-motion tokens, with a 3D U-Net detokenizer conditioned on motion for pixel generation and a noise-inversion constraint for long videos.
Key results
- Achieves 80.3 on VQAv2, 56.0 on VizWiz, and 1551.8 on MME among 7B models in image understanding, leading on three of four multimodal benchmarks.
- Zero-shot MSVD-QA accuracy 73.2 vs 70.7 for Video-LLaVA; ActivityNet-QA accuracy 50.1 vs 45.3.
- Text-to-video on MSR-VTT FVD 188.36 (best among compared, public-data group) and UCF-101 IS 44.26 with 10M training clips.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| include motion tokens (ablation) | Video-LaVIT · MSVD-QA | accuracy: 67.3 -> 73.2 | Helped |
| include motion tokens (ablation) | Video-LaVIT · ActivityNet-QA | accuracy: 47.4 -> 50.1 | Helped |
| include motion tokens (ablation) | Video-LaVIT · UCF-101 | IS: 29.56 -> 44.26 | Helped |
| include motion tokens (ablation) | Video-LaVIT · UCF-101 | FVD: 442.80 -> 280.57 | Helped |
| use 135 rather than 256 motion tokens (ablation) | Video-LaVIT · MSVD-QA | accuracy: 69.2 -> 73.2 | Helped |
| use 135 rather than 256 motion tokens (ablation) | Video-LaVIT · UCF-101 | FVD: 281.24 -> 280.57 | Helped |
| include motion tokens in image comprehension (ablation) | Video-LaVIT · VQA v2 | score: 80.0 -> 80.3 | Helped |
| include motion tokens in image comprehension (ablation) | Video-LaVIT · ScienceQA-IMG | score: 71.5 -> 70.0 | Hurt |
| include motion tokens in image comprehension (ablation) | Video-LaVIT · MM-Vet | score: 34.5 -> 33.2 | Hurt |
| remove svd-img2vid-xt weight initialization (ablation) | Video-LaVIT text-to-video generation · MSR-VTT | CLIPSIM: 0.3010 -> 0.3012 | No clear effect |
| remove svd-img2vid-xt weight initialization (ablation) | Video-LaVIT text-to-video generation · MSR-VTT | FVD: 169.51 -> 188.36 | Hurt |
| remove svd-img2vid-xt weight initialization (ablation) | Video-LaVIT text-to-video generation · UCF-101 | IS: 37.96 -> 44.26 | Helped |
Surprising findings
- Adding motion tokens slightly lowers some image benchmarks (e.g., SQA 71.5 without motion vs 70.0 with, MMB 67.5 vs 67.3), per the appendix ablation.
- Initializing the detokenizer from SVD img2vid-xt gave a better UCF-101 FVD but a lower IS and mixed MSR-VTT results (FVD 169.51 vs 188.36, FID 11.80 vs 11.27).
Limitations
- Keyframes are fixed to MPEG-4 I-frames; more sophisticated keyframe selection is not explored.
- Video generation quality is not much far from Gen-2 and lags on some metrics (e.g., MSR-VTT CLIPSIM and FID vs best baselines), and long-video consistency relies on a noise-inversion heuristic.
- Model is not aligned with safety procedures; the paper notes hallucination and misinformation risks.
Experimental setup
Datasets: WebVid-10M InternVid-14M-aesthetics subset Conceptual Captions SBU BLIP-Capfilt RedPajama LLaVA v1.5 instruction data Video-ChatGPT instruction data MS-COCO MSR-VTT UCF-101 EvalCrafter prompts
Benchmarks: VQA v2 GQA VizWiz ScienceQA-IMG MME MMBench SEED MM-Vet MSVD-QA MSRVTT-QA ActivityNet-QA Perception Test
Metrics: accuracy relative score CLIPSIM FVD FID IS KVD
Baselines: Video-LLaVA LLaMA-VID LLaVA-1.5 VideoChat2 InternVideo Gen-2 SVD-XT VideoPoet CogVideo FreeNoise
Builds on: LaVIT Stable Video Diffusion VQ-VAE EDM DDIM
Closest papers by meaning
- LARP: Tokenizing Videos with a Learned Autoregressive Generative Prior (ICLR 2025 · similarity 0.91)
- Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization (ICLR 2024 · similarity 0.91)
- Unleashing Hour-Scale Video Training for Long Video-Language Understanding (NeurIPS 2025 · similarity 0.90)
- Language Model Beats Diffusion - Tokenizer is key to visual generation (ICLR 2024 · similarity 0.90)
- V-CAST: Video Curvature-Aware Spatio-Temporal Pruning for Efficient Video Large Language Models (NeurIPS 2026 · similarity 0.89)
- VideoPoet: A Large Language Model for Zero-Shot Video Generation (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.