Research map / Vision-language and multimodal / Human and embodied modeling
HoVLE: Unleashing the Power of Monolithic Vision-Language Models with Holistic Vision-Language Embedding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes HoVLE, a monolithic VLM with a holistic embedding module (8 causal Transformer layers) trained by distilling InternViT features and LLM text embeddings on unpaired random images/tokens, then aligned and instruction-tuned on top of a frozen-then-unfrozen InternLM2-1.8B.
Problem
Monolithic vision-language models that feed image and text tokens directly into LLMs lag far behind compositional VLMs, and tuning pre-trained LLMs for vision may degrade their language abilities.
Approach
Insert a trainable holistic embedding module (causal Transformer layers over patch and text embeddings) that maps images and text into a shared space, trained first via cosine-similarity distillation from a pre-trained vision encoder and LLM embedding layer using unpaired random images and random text tokens, so the LLM itself need not be tuned for vision during alignment.
Key results
- HoVLE (2.6B) reaches 71.9 on MMBench and 1864 on MME, about 15 points above the best prior monolithic model on MMBench.
- HoVLE (HD) reaches 79.7-level-competitive results on several VQA tasks, e.g., 84.7 DocVQA and 52.1 InfoVQA for HoVLE vs. 22.0 DocVQA for EVE-7B.
- Ablation: removing the distillation stage causes the largest drop (MMB 72.0 to 37.3 without distillation... see Table 4 for stage ablation), and random text inputs for distillation outperform paired image-text inputs (MMB 72.0 vs 71.7).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add holistic embedding module (8 causal Transformer layers) vs none (0 layers) (ablation) | HoVLE · MMBench / TextVQA / DocVQA ablation | MMB score: 35.4 -> 72.0 MMB at 8 layers | Helped |
| increase holistic embedding depth from 4 to 12 layers (ablation) | HoVLE · MMBench ablation | MMB score: 67.8 (4 layers) -> 72.3 (12 layers) | Helped |
| remove distillation stage (ablation) | HoVLE training · MMBench ablation | MMB score: 72.0 -> 37.3 | Hurt |
| remove alignment stage (ablation) | HoVLE training · MMBench ablation | MMB score: 72.0 -> 66.1 | Hurt |
| remove instruction tuning stage (ablation) | HoVLE training · MMBench ablation | MMB score: 72.0 -> 71.3 | Hurt |
| use random text tokens instead of paired image-text text inputs during distillation (ablation) | HoVLE distillation · MMBench ablation | MMB score: 71.7 -> 72.0 | Helped |
| increase distillation data from 300M to 500M images (ablation) | HoVLE distillation · MMBench / POPE / TextVQA / DocVQA | score | Helped |
| use higher-resolution input (HD variant with pixel shuffle) | HoVLE · VQA benchmarks (TextVQA, DocVQA, InfoVQA) | score: TextVQA 66.0 -> 70.9; DocVQA 84.7 -> 86.1 | Helped |
| holistic embedding (monolithic HoVLE) vs prior monolithic VLMs | monolithic VLM design · MMBench | MMB score: ~15 points over previous best monolithic (EVE-7B HD 52.3 vs HoVLE 71.9 in table) | Helped |
Surprising findings
- Random unpaired image and random text tokens for distillation outperform paired LAION image-text inputs.
Limitations
- Experiments limited to the 2B parameter scale due to compute.
- Still trails leading compositional VLMs such as Qwen2-VL-2B and Phi-3 Vision.
Experimental setup
Datasets: LAION-2B alignment mixture (captioning, detection, OCR; 50M) instruction tuning mixture (VQA, math, knowledge; 5M)
Benchmarks: MMBench MME MMMU MM-Vet POPE SEED MathVista HallusionBench CCBench TextVQA ChartQA InfoVQA
Metrics: accuracy score
Baselines: InternVL2 Qwen2-VL-2B Phi-3 Vision EVE-7B Emu3 Chameleon-7B SOLO Fuyu-8B
Builds on: InternVL InternLM2 EVE SOLO
Closest papers by meaning
- From Pixels to Words -- Towards Native Vision-Language Primitives at Scale (ICLR 2026 · similarity 0.88)
- DH-Set: Improving Vision-Language Alignment with Diverse and Hybrid Set-Embeddings Learning (CVPR 2025 · similarity 0.87)
- Large Vision-Language Models Get Lost in Attention (ICML 2026 · similarity 0.87)
- VL-SAE: Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Set (NeurIPS 2025 · similarity 0.86)
- Language-Instructed Vision Embeddings for Controllable and Generalizable Perception (ICLR 2026 · similarity 0.86)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.