Research map / Vision-language and multimodal / Cross-modal learning
mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces mPLUG-Owl2, an MLLM with a modality-adaptive module that shares FFN/query projections across vision and text while separating key/value projections and layer norms, trained with two-stage vision-language pretraining and joint text+multimodal instruction tuning.
Problem
Multi-modal LLMs trained with a single shared module suffer modality interference, degrading text performance, while frozen-LLM designs limit modality collaboration.
Approach
A modality-adaptive module inside the language decoder uses shared query and FFN parameters but modality-specific key, value, and layer norm parameters, plus a visual abstractor and joint text-and-multimodal instruction tuning with a trainable vision encoder, so that vision and language collaborate without interfering.
Key results
- Achieves 79.4 on VQAv2 and 54.3 zero-shot TextVQA at 448x448 resolution, with state-of-the-art Flickr30K CIDEr of 85.1 among generalists.
- Reaches MMLU 53.4 and BBH 45.0 on pure-text benchmarks, improving over LLaMA-2-7B-based baselines such as Vicuna-v1.5 (51.1 MMLU, 41.2 BBH).
- Ablation shows the modality-adaptive module improves MMLU from 52.8 to 53.4-type gains and recovers text degradation (about 5.7% drop with multimodal-only tuning).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the Modality-Adaptive Module (ablation) | mPLUG-Owl2 · VQAv2 | score: 76.2 -> 76.8 | Helped |
| add the Modality-Adaptive Module (ablation) | mPLUG-Owl2 · MMLU | score: 51.6 -> 52.8 | Helped |
| jointly use text-only and multi-modal instruction data (ablation) | mPLUG-Owl2 · MMLU | score: 45.4 -> 51.6 | Helped |
| add multi-modal instruction data to text instruction data (ablation) | mPLUG-Owl2 · VQAv2 | score: 58.2 -> 76.3 | Helped |
| unfreeze the vision encoder during instruction tuning (ablation) | mPLUG-Owl2 · VQAv2 | score: 74.8 -> 76.2 (+1.4) | Helped |
| unfreeze the vision encoder during instruction tuning (ablation) | mPLUG-Owl2 · MMBench | score: 63.8 -> 62.7 (-1.1) | Hurt |
| use layer-wise learning rate decay for the vision encoder (ablation) | mPLUG-Owl2 with trainable vision encoder · TextVQA | score: 40.3 -> 42.5 (+2.7) | Helped |
| increase learnable queries from 8 to 64 (ablation) | mPLUG-Owl2 visual abstractor · VQAv2 | score: 58.3 -> 76.8 | Helped |
| increase learnable queries from 64 to 128 (ablation) | mPLUG-Owl2 visual abstractor · VQAv2 | score: 76.8 -> 76.7 | No clear effect |
| increase image resolution from 224 × 224 to 448 × 448 (ablation) | mPLUG-Owl2 · TextVQA | score: 42.5 -> 54.3 (+11.8) | Helped |
| increase image resolution from 224 × 224 to 448 × 448 (ablation) | mPLUG-Owl2 · VQAv2 | score: 76.8 -> 79.4 (+2.6) | Helped |
| increase image resolution from 224 × 224 to 448 × 448 (ablation) | mPLUG-Owl2 · MM-Vet | score: 34.0 -> 36.2 (+2.2) | Helped |
Surprising findings
- Multimodal-only instruction tuning degrades text benchmark performance by about 5.7%, and joint text+multimodal tuning with the modality-adaptive module improves both multimodal and text scores.
Limitations
- Text-only gains measured on a limited set of benchmarks; modality collaboration mechanism is analyzed qualitatively via attention maps.
- Performance on MME is lower than some baselines, attributed to limited test sample size.
Experimental setup
Datasets: CC3M CC12M COCO Laion-en COYO DataComp TextCaps VQAv2 OKVQA OCR-VQA GQA A-OKVQA
Benchmarks: COCO Caption Flickr30K VQAv2 OKVQA GQA VizWizQA TextVQA SciQA (IMG) MME MMBench MM-Vet SEED-Bench
Metrics: CIDEr accuracy score exact match
Baselines: BLIP-2 InstructBLIP Qwen-VL-Chat LLaVA-1.5 MiniGPT-4 mPLUG-Owl Shikra Video-ChatGPT LLaMA-2-Chat Vicuna-v1.5
Builds on: mPLUG-Owl BLIP-2 LLaVA LLaMA-2 Perceiver
Closest papers by meaning
- mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models (ICLR 2025 · similarity 0.88)
- Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.88)
- HaploVL: A Single-Transformer Baseline for Multi-Modal Understanding (ICML 2025 · similarity 0.87)
- On Path to Multimodal Generalist: General-Level and General-Bench (ICML 2025 · similarity 0.87)
- Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models (NeurIPS 2026 · similarity 0.87)
- Can Large Language Models Help Multimodal Language Analysis? MMLA: A Comprehensive Benchmark (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.