Research map / Vision-language and multimodal / Multimodal LLMs
mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces mPLUG-Owl3, an 8B MLLM that uses sparsely inserted Hyper Attention blocks (parallel cross-attention to image features, with MI-Rope and adaptive gating) to handle long image sequences, interleaved inputs, and long videos efficiently.
Problem
Existing multimodal LLMs that insert visual tokens into the text sequence or use Flamingo-style cross-attention struggle with long image sequences, due to context-window cost, memory/latency overhead, and loss of fine-grained visual information.
Approach
Replace a few transformer blocks of the LLM with Hyper Attention Transformer Blocks that run cross-attention to visual features in parallel with self-attention, reusing the self-attention query, initializing visual K/V from the LLM's KV projection, adding an adaptive text-conditioned gate, and using multimodal-interleaved RoPE (MI-Rope) to encode image positions.
Key results
- mPLUG-Owl3 (8B) reports 53.5 on VideoMME and 52.1 on LongVideoBench-val, significantly above the compared 8B video models.
- On the Distractor Resistance test, mPLUG-Owl3 drops only to 43.09% at 50 images while LLaVA-Next-Interleave falls to 12.52%.
- mPLUG-Owl3 reports 82.1 on VQAv2, 65.0 on GQA and 63.5 on VizWizQA among 8B-level models, and 90.8 on NLVR2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Hyper Attention instead of concatenating image and text features (ablation) | mPLUG-Owl3 ablation model · NLVR2 | overall score: 55.7 -> 59.5 | Helped |
| Use Hyper Attention instead of concatenating image and text features (ablation) | mPLUG-Owl3 ablation model · GQA | overall score: 59.0 -> 57.6 | Hurt |
| Place cross-attention before self-attention rather than use Hyper Attention (ablation) | mPLUG-Owl3 ablation model · MvBench | overall score: 42.8 -> 43.0 | Helped |
| Place cross-attention after self-attention rather than use Hyper Attention (ablation) | mPLUG-Owl3 ablation model · Mantis-Eval | overall score: 47.0 -> 51.6 | Hurt |
| Use eight Hyper Attention layers instead of four (ablation) | mPLUG-Owl3 ablation model · NLVR2 | overall score: 59.5 -> 52.4 | Hurt |
| Use two Hyper Attention layers instead of four (ablation) | mPLUG-Owl3 ablation model · NLVR2 | overall score: 59.5 -> 58.3 | Hurt |
| Add adaptive gating (ablation) | basic Hyper Attention · GQA | overall score: 53.3 -> 55.7 | Helped |
| Add adaptive gating (ablation) | basic Hyper Attention · Mantis | overall score: 41.9 -> 47.9 | Helped |
| Add shared LayerNorm (ablation) | Hyper Attention with adaptive gating · GQA | overall score: 55.7 -> 58.1 | Helped |
| Add shared LayerNorm (ablation) | Hyper Attention with adaptive gating · VideoMME | overall score: 40.1 -> 38.4 | Hurt |
| Add MI-Rope (ablation) | Hyper Attention with adaptive gating and shared LayerNorm · NLVR2 | overall score: 54.9 -> 59.5 | Helped |
| Add MI-Rope (ablation) | Hyper Attention with adaptive gating and shared LayerNorm · Mantis | overall score: 46.1 -> 51.6 | Helped |
Surprising findings
- The concatenation-based baseline performs best on single-image tasks but degrades sharply on multi-image and video settings.
- Using only four Hyper Attention layers works better than eight, which hurt both single-image and zero-shot generalization.
Limitations
- The vision encoder is frozen, limiting fine-grained low-level perception (e.g., BLINK) and text-rich performance (AI2D).
- Performance drops substantially with many distractor images (28.58% at 400 images) since training data has only about 6-8 images per sample.
- Hallucinations observed in long video qualitative examples.
Experimental setup
Datasets: DataComp-1B LAION COYO-700M CC12M ShareGPTVideo LLaVA-SFT-665K The Cauldron Mantis LLaVA-Interleave MMDU VATEX ALLaVA
Benchmarks: VQAv2 OK-VQA GQA VizWizQA TextVQA MMBench MM-Vet POPE AI2D NextQA MVBench VideoMME
Metrics: accuracy CircularEval accuracy
Baselines: LLaVA-Next-Interleave Mantis-Idefics2 Idefics2 Mantis-SigLIP VILA-1.5 GPT-4o GPT-4V CogVLM EVLM-Chat
Builds on: Flamingo LLaVA-1.5 Qwen2 SigLIP
Closest papers by meaning
- MMInference: Accelerating Pre-filling for Long-Context Visual Language Models via Modality-Aware Permutation Sparse Attention (ICML 2025 · similarity 0.89)
- Dynamic-LLaVA: Efficient Multimodal Large Language Models via Dynamic Vision-language Context Sparsification (ICLR 2025 · similarity 0.89)
- MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Models (CVPR 2025 · similarity 0.88)
- Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.88)
- LongVILA: Scaling Long-Context Visual Language Models for Long Videos (ICLR 2025 · similarity 0.88)
- mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.