Research map / Video / Video-language understanding
LLaVA-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LLaVA-NeXT-Interleave, a single LMM trained on the interleaved image-text format (M4-Instruct, 1177.6K samples) to unify multi-image, video, 3D, and single-image tasks, with a new LLaVA-Interleave Bench for evaluation.
Problem
Open LMMs focus on single-image tasks and handle multi-image, video, and 3D scenarios with separate task-specific models, so they lack a unified framework and cross-scenario generalization.
Approach
Treat the image-text interleaved format as a general template: video becomes multi-frame, 3D becomes multi-view, and single images become multi-patch, so one model can be instruction-tuned jointly on all four scenarios, starting from a single-image LLaVA-NeXT checkpoint.
Key results
- The 14B model reaches 62.3 average on LLaVA-Interleave Bench in-domain and 44.3 out-domain, versus GPT-4V's 39.2 and 57.8 respectively.
- With DPO, the 7B model achieves SoTA on the VDD and VideoChat-GPT benchmarks, surpassing LLaVA-NeXT-Video (34B) on those metrics.
- The 14B model reaches 59.2 average on 3D benchmarks, above GPT-4V (34.6) and 3D-LLM (22.9).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Continue training from a single-image model rather than stage-1 pre-training (ablation) | LLaVA-NeXT-Interleave · Multi-image Mantis | score: 41.0 -> 45.6 | Helped |
| Continue training from a single-image model rather than stage-1 pre-training (ablation) | LLaVA-NeXT-Interleave · Multi-image NLVR2 | score: 54.0 -> 67.8 | Helped |
| Do not pool video vision tokens instead of pooling to 1/4 (ablation) | LLaVA-NeXT-Interleave · Video VDD | score: 3.38 -> 3.46 | Helped |
| Increase inference frames from 10 to 16 without pooling (ablation) | LLaVA-NeXT-Interleave · Video ActivityNet-QA | accuracy: 52.9 -> 54.4 | Helped |
| Add video DPO training (ablation) | LLaVA-NeXT-Interleave · Video VDD | score: 3.57 -> 3.90 | Helped |
| Add video DPO training (ablation) | LLaVA-NeXT-Interleave · Multi-view ScanQA | score: 32.2 -> 25.5 | Hurt |
| Add video DPO training (ablation) | LLaVA-NeXT-Interleave · Multi-image Q-Bench | score: 73 -> 86.8 | Helped |
| Combine multi-image, video, 3D, and single-image training data (ablation) | LLaVA-NeXT-Interleave · Video benchmarks | performance | Helped |
Surprising findings
- Initializing from a single-image checkpoint outperforms initializing from a stage-1 pre-trained checkpoint on multi-image, video, and 3D tasks.
- Video DPO improves video results but slightly degrades some multi-image and single-image tasks.
Limitations
- Video DPO slightly hurts performance on other tasks (e.g., Mantis, ScanQA, DocVQA drop after video DPO).
- Multi-image gains on some benchmarks (e.g., MMMU-mv) are modest; 7B lags GPT-4V on some tasks.
- Some emerging capabilities are only shown qualitatively with a few examples.
Experimental setup
Datasets: M4-Instruct LLaVA-Hound (ShareGPTVideo) subset NExT-QA STAR nuScenes QA ALFRED ScanQA 3D-LLM LLaVA-NeXT SFT data (40%) HQ-Edit MagicBrush NLVR2
Benchmarks: LLaVA-Interleave Bench Mantis-Eval BLINK MMMU-mv NExT-QA MVBench ActivityNet-QA Video Detailed Description VideoChat-GPT ScanQA 3D-LLM nuScenes VQA
Metrics: accuracy GPT score average score
Baselines: GPT-4V LLaVA-NeXT-Image VPG-C Mantis VideoChatGPT Video-LLaVA VISTA-LLaMA VideoChat2 LLaMA-VID LLaVA-NeXT-Video 3D-LLM Point-Bind & LLM
Builds on: LLaVA-NeXT Mantis LLaVA-Hound Qwen 1.5 SigLIP
Closest papers by meaning
- Improved Baselines with Visual Instruction Tuning (CVPR 2024 · similarity 0.93)
- LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Token (ICLR 2025 · similarity 0.92)
- iLLaVA: An Image is Worth Fewer Than 1/3 Input Tokens in Large Multimodal Models (ICLR 2026 · similarity 0.90)
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (NeurIPS 2024 · similarity 0.90)
- LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning (CVPR 2026 · similarity 0.90)
- Apollo: An Exploration of Video Understanding in Large Multimodal Models (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.