Research map / Vision-language and multimodal / Multimodal LLMs
Unveiling Encoder-Free Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds EVE, an encoder-free decoder-only vision-language model trained from Vicuna-7B with a lightweight patch embedding layer and patch-aligning supervision from CLIP-ViT-L/14, using 35M public images.
Problem
Vision-language models rely on vision encoders that impose fixed resolution/aspect-ratio priors and cost deployment efficiency; training pure encoder-free VLMs is slow to converge and lags encoder-based models.
Approach
Train a decoder-only VLM initialized from an LLM with a lightweight patch embedding layer, an LLM-guided pre-alignment stage with frozen LLM, and hierarchical multi-layer MSE alignment of patch features to a CLIP vision encoder (used only in training) to inject fine-grained visual recognition.
Key results
- EVE-7B (HD) reaches 78.6 on VQA-v2, 62.6 on GQA, and 56.8 on SEED, outperforming Fuyu-8B (HD) on all reported benchmarks (e.g., POPE 85.0 vs 74.1, MME 1305.7 vs 728.6).
- EVE-7B (HD) is competitive with encoder-based VLMs such as LLaVA-1.5 and InternVL-Chat while using only 35M public samples.
- Vision-encoder-free EVE-7B has about 42 GFLOPs vision cost vs 372 for LLaVA-1.5 and 170 vs 1860 for EVE-7B (HD) vs LLaVA-1.6 (HD), with much lower latency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use high-resolution SFT and additional SFT data (EVE-7B → EVE-7B (HD)) | EVE-7B · VizWiz | accuracy: 41.8 -> 51.1 | Helped |
| Remove PEL CA1 (ablation) | EVE₀.₅M · VQA-v2 | accuracy: 58.5 -> 58.2 | Hurt |
| Remove PEL CA2 (ablation) | EVE₀.₅M · VQA-v2 | accuracy: 58.5 -> 58.4 | Hurt |
| Replace pairwise patch alignment with next-patch alignment (ablation) | EVE₀.₅M · VQA-v2 | accuracy: 58.5 -> 58.0 | Hurt |
| Add PAL supervision (ablation) | EVE₀.₅M · VQA-v2 | accuracy: 55.6 -> 58.5 | Helped |
| Add PAL supervision (ablation) | EVE₄M · VQA-v2 | accuracy: 66.4 -> 69.4 | Helped |
| Add PAL supervision (ablation) | EVE₈M · VQA-v2 | accuracy: 69.4 -> 71.2 | Helped |
| Add Stage 1 LLM-guided pre-alignment before Stage 2 (ablation) | EVE · VQA-v2 | accuracy: 50.2 -> 71.2 | Helped |
| Increase Stage 2 pre-training from 4M to 8M without Stage 1 (ablation) | EVE w/o Stage 1 · VQA-v2 | accuracy: 64.6 -> 50.2 | Hurt |
| Increase Stage 2 pre-training from 4M to 8M after Stage 1 (ablation) | EVE w/ Stage 1 · VQA-v2 | accuracy: 69.4 -> 71.2 | Helped |
| Add Stage 3 SFT (665K) after Stage 1-2 (ablation) | EVE · VQA-v2 | accuracy: 71.2 -> 75.4 | Helped |
| Add high-resolution training (HD) after Stage 3 (665K) (ablation) | EVE · VQA-v2 | accuracy: 75.4 -> 77.5 | Helped |
Surprising findings
- Direct training without an LLM-guided pre-alignment stage collapses as data grows, even with vision supervision.
- Training solely on vision-language data reduces language capability (SQA drops from 65.3% to 63.0%), indicating catastrophic forgetting.
Limitations
- Remaining performance gap to state-of-the-art encoder-based VLMs, especially on MME, MMBench, SQA and MM-Vet.
- Weak instruction following for option letters and binary questions, and degraded language ability after vision-language-only training.
- Compared with encoder-based models only at ~7B scale with LLM-matched training; encoder-free performance at equal LLM capacity and data is untested.
Experimental setup
Datasets: SA-1B OpenImages LAION LLaVA-mix-665K Bunny-695K Vision-Flan AI2D Synthdog DVQA ChartQA DocVQA
Benchmarks: VQA-v2 GQA VizWiz ScienceQA-IMG TextVQA POPE MME MMBench SEED-Bench MM-Vet
Metrics: accuracy score
Baselines: Fuyu-8B LLaVA-1.5 LLaVA-1.6 InstructBLIP IDEFICS-9B QwenVL-Chat InternVL-Chat mPLUG-Owl2
Builds on: LLaVA-1.5 Vicuna CLIP Fuyu
Closest papers by meaning
- Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs (NeurIPS 2026 · similarity 0.92)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.91)
- From Pixels to Words -- Towards Native Vision-Language Primitives at Scale (ICLR 2026 · similarity 0.91)
- Rethinking Model Selection in VLM Through the Lens of Gromov-Wasserstein Distance (CVPR 2026 · similarity 0.90)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.90)
- Eve: Efficient Multimodal Vision Language Models with Elastic Visual Experts (AAAI 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.