Research map / Vision-language and multimodal / Multimodal LLMs
Eve: Efficient Multimodal Vision Language Models with Elastic Visual Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Eve builds a 1.8B-parameter efficient VLM by training a vision adapter and LoRA-tuned LLM in two stages, then adding frozen-language-FFN plus trainable elastic vision FFN experts with token-allocation routing in a third stage to boost multimodal ability while preserving language skills.
Problem
Efficient small vision language models often sacrifice linguistic ability for multimodal gains or need extensive training; deploying capable VLMs on edge devices requires preserving language competence at small scale.
Approach
Adds a vision-specific FFN expert (Elastic Vision FFN) alongside a frozen language FFN in alternating LLM layers, trained alone with a routing layer and an Img-GBPR token allocation scheme that prioritizes modality-matched tokens, so the original language path is untouched and language ability is preserved.
Key results
- Eve-VLM (1.5B LLM, SigLIP, 384px) reaches 68.87% VLM benchmark average, the best among models under 3B activated parameters and above DeepSeek-VL (66.9-ish, reported as 1.9 points lower) and LLaVA-1.5 7B (68.79%).
- On language benchmarks, Eve scores 58.56 average versus 46.08 for LLaVA-Phi-2.7B and 45.70 for DeepSeek-VL-1.3B.
- In ablation, the EVF layer keeps language average at 58.65 (same as Stage 2) while the MoE alternative drops it to 55.03; Img-GBPR raises ResNet50-based VLM average from 53.83 to 54.92.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Stage 2 with Stage 3 MoE layers (ablation) | PanGu-π-1.5B VLM · VLM benchmarks | VLM AVG: 61.23 -> 62.23 | Helped |
| Replace Stage 2 with Stage 3 MoE layers (ablation) | PanGu-π-1.5B VLM · Language benchmarks | Language AVG: 58.65 -> 55.03 | Hurt |
| Add EVF layers in Stage 3 (ablation) | PanGu-π-1.5B VLM · VLM benchmarks | VLM AVG: 61.23 -> 61.93 | Helped |
| Add EVF layers in Stage 3 (ablation) | PanGu-π-1.5B VLM · Language benchmarks | Language AVG: 58.65 -> 58.65 | No clear effect |
| Replace random token allocation with GBPR (ablation) | Stage 3 EVF · VLM benchmarks | VLM AVG: 61.93 -> 62.37 | Helped |
| Replace random token allocation with Img-GBPR (ablation) | Stage 3 EVF · VLM benchmarks | VLM AVG: 61.93 -> 62.48 | Helped |
| Replace MobileLLaMA with PanGu-π-1.5B (ablation) | ResNet50 VLM baseline · VLM benchmarks | VLM AVG: 51.97 -> 53.36; 1.4% | Helped |
| Add EVF layers (ablation) | ResNet50 + PanGu-π-1.5B · VLM benchmarks | VLM AVG: 53.36 -> 53.83 | Helped |
| Add Img-GBPR (ablation) | ResNet50 + PanGu-π-1.5B + EVF · VLM benchmarks | VLM AVG: 53.83 -> 54.92 | Helped |
| Replace ResNet50 with SigLip-L (ablation) | PanGu-π-1.5B + EVF + Img-GBPR · VLM benchmarks | VLM AVG: 54.92 -> 63.03; 8% | Helped |
| Replace PanGu-π-1.5B with PanGu-π-1.5B-Pro (ablation) | SigLip-L + EVF + Img-GBPR · VLM benchmarks | VLM AVG: 63.03 -> 64.52; 1.5% | Helped |
| Use curated Stage 3 data (ablation) | SigLip-L + PanGu-π-1.5B-Pro + EVF + Img-GBPR · VLM benchmarks | VLM AVG: 64.52 -> 68.87; 4.3% | Helped |
Surprising findings
- The MoE-style alternative improved multimodal accuracy slightly more than EVF (62.23 vs 61.93) but cost about 3.6 points of language accuracy.
- Vision encoder pre-training data choice (ImageNet-1K, ImageNet-22K, LAION-400M) changed VLM average by only about 0.3 points in a narrow range, suggesting the language side is largely insensitive to the encoder.
Limitations
- Findings on language-model competence and training data effects may not transfer to newer language models, per the authors.
- Ablations are largely run with ResNet-50 and PanGu-pi-1.5B and a single run, and the best configuration differs from the ablation setup.
- Broader impacts: training on internet-sourced data risks absorbing and amplifying biases.
Experimental setup
Datasets: LLaVA-558k LLaVA-mixed-665k ShareGPT4V LAION-GPTV VFLAN ScienceQA IconQA TextVQA VSR OKVQA VQAv2 KVQA
Benchmarks: GQA ScienceQA-IMG TextVQA POPE MME MMBench C-Eval CMMLU MMLU BoolQ PIQA EPRSTMT
Metrics: accuracy weighted average accuracy
Baselines: LLaVA-1.5 LLaVA-1.6 TinyGPT-V Mini-Gemini MobileVLM MobileVLM v2 LLaVA-Phi MoE-LLaVA DeepSeek-VL
Builds on: MoE-LLaVA MobileVLM LLaVA-1.5 PanGu-pi
Closest papers by meaning
- EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE (AAAI 2024 · similarity 0.91)
- Unveiling Encoder-Free Vision-Language Models (NeurIPS 2024 · similarity 0.90)
- What matters when building vision-language models? (NeurIPS 2024 · similarity 0.89)
- Are We on the Right Way for Evaluating Large Vision-Language Models? (NeurIPS 2024 · similarity 0.89)
- Mordal: Automated Pretrained Model Selection for Vision Language Models (ICLR 2026 · similarity 0.89)
- A Comprehensive Overhaul of Multimodal Assistant with Small Language Models (AAAI 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.