Research map / Vision-language and multimodal / Multimodal LLMs
Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Mono-InternVL, a monolithic MLLM that embeds visual experts (multimodal MoE) into a frozen pre-trained LLM and pre-trains them via a three-stage Endogenous Visual Pre-training (EViP) curriculum from noisy to high-quality data.
Problem
Monolithic MLLMs trained by continuous pre-training suffer catastrophic forgetting and unstable optimization, leaving them behind modular MLLMs in performance.
Approach
Embed a separate visual expert FFN set (initialized from the LLM's FFNs) via a static multimodal MoE and freeze the LLM during visual pre-training (delta tuning), so visual learning does not overwrite language knowledge; progressively pre-train visual experts from noisy to high-quality data.
Key results
- Mono-InternVL-2B outperforms InternVL-1.5-2B by +0.8% on average over 9 MLLM benchmarks and gains +113 points on OCRBench.
- Mono-InternVL reduces first-token latency by up to 67% versus InternVL-1.5 with LMDeploy, raising throughput about 31%.
- Average performance of InternVL-1.5-2B exceeds monolithic SoTA EVE-7B (HD) by +15.5%, while Mono-InternVL-2B (1.8B activated) beats EVE-7B by a wide margin.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Mono-InternVL instead of InternVL-1.5 | Mono-InternVL-2B · 6 multimodal benchmarks | average benchmark performance: +0.8% | Helped |
| use Mono-InternVL instead of InternVL-1.5 | Mono-InternVL-2B · InfoVQA | score: -12.4% | Hurt |
| use Mono-InternVL instead of InternVL-1.5 | Mono-InternVL · OCRBench | score: +113 points | Helped |
| use Mono-InternVL instead of InternVL-1.5 | Mono-InternVL · inference | first token latency: up to -67% | Helped |
| use Mono-InternVL instead of InternVL-1.5 | Mono-InternVL · inference | throughput: around 31% | Helped |
| add visual experts to full tuning (ablation) | Mono-InternVL · GQA | performance: +1.6% | Helped |
| use delta tuning instead of full tuning (ablation) | Mono-InternVL · SQA-I | performance: +18.8% | Helped |
| use delta tuning instead of full tuning (ablation) | Mono-InternVL · AI2D | performance: 16.1% | Helped |
| unfreeze multi-head attention layers during alignment learning (ablation) | Mono-InternVL · alignment-learning evaluation metrics | performance | Helped |
| add semantic learning and alignment learning after concept learning (ablation) | Mono-InternVL · downstream tasks | performance as pre-training data increases | Helped |
Surprising findings
- Mono-InternVL trails InternVL-1.5 substantially on high-resolution InfoVQA (-12.4%).
- Full tuning of the LLM during visual pre-training yields the worst downstream performance, while adding visual experts without freezing still helps.
Limitations
- Inferior to modular InternVL-1.5 on high-resolution benchmarks such as InfoVQA
- Relies on large-scale noisy and synthetic pre-training data (over 1B samples) and 256 A100 GPUs
Experimental setup
Datasets: LAION-2B COYO-700M InternVL-1.5 pre-training data LLaVA-665k bilingual instruction data (~5M)
Benchmarks: MMBench-EN MMVet MMMU MME MathVista SEED Image OCRBench HallusionBench CCBench TextVQA SQA GQA
Metrics: accuracy CIDEr first token latency throughput
Baselines: InternVL-1.5 EVE-7B (HD) Emu3 Chameleon-7B MM1 Fuyu-8B SOLO
Builds on: InternVL-1.5 InternLM2 delta tuning VL-MoE
Closest papers by meaning
- Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models (CVPR 2026 · similarity 0.91)
- LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning (CVPR 2026 · similarity 0.90)
- Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models (CVPR 2026 · similarity 0.90)
- NaViL: Rethinking Scaling Properties of Native Multimodal Large Language Models under Data Constraints (NeurIPS 2025 · similarity 0.90)
- MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning (ICLR 2025 · similarity 0.90)
- CoVFT: Context-aware Visual Fine-tuning for Multimodal Large Language Models (CVPR 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.