Research map / Large language models / Language, safety and interpretability
HaploVL: A Single-Transformer Baseline for Multi-Modal Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a single-transformer early-fusion LMM (HaploVL) whose pre-decoder is initialized from CLIP-ViT-L and distilled with feature and current-token losses before instruction tuning on a Llama-3 or Qwen2.5 LLM.
Problem
Native single-transformer multi-modal models are resource-intensive to train from scratch and still lag compositional encoder-decoder LMMs on multi-modal benchmarks, especially fine-grained perception.
Approach
Early-fusion model that feeds raw linear-patch vision embeddings and text embeddings into a pre-decoder initialized from CLIP-ViT-L (distilled via vision and text feature losses plus a current-token prediction loss), followed by a post-decoder inherited from an LLM, so that prior vision and language knowledge reduces data and compute needs.
Key results
- HaploVL-7B-Pro reaches 75.0 on SEED and 88.7 on POPE, with 50.1 on MMVP, while outperforming Emu3-8B on MMBench by 15.1% per the authors.
- Under matched Vicuna-7B and 665K data, HaploVL-7B beats LLaVA-1.5-7B by 3.4% on MMVP and 4.2% on MMStar, and gains 4.9% on MMStar fine-grained perception.
- Llama-3-8B base raises the average from 61.3% to 64.2% in the ablation, and removing the modal expansion stage drops performance by 4.3%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use early fusion with one transformer instead of a separate vision encoder and LLM | HaploVL-7B with Vicuna-7B, compared with LLaVA-1.5-7B · MMVP | score: 21.3 -> 24.7 | Helped |
| Use early fusion with one transformer instead of a separate vision encoder and LLM | HaploVL-7B with Vicuna-7B, compared with LLaVA-1.5-7B · MMStar | average score: 30.3 -> 34.5 | Helped |
| Replace Vicuna-7B with Llama-3-8B (ablation) | HaploVL · multiple benchmarks (average) | average score: +2.5% | Helped |
| Increase input resolution from 336 × 336 to 672 × 672 (ablation) | HaploVL with Llama-3-8B and 665K instruction data · multiple benchmarks (average) | average score: +3.3% | Helped |
| Increase visual instruction data from 665K to 4M (ablation) | HaploVL with Llama-3-8B at 672 × 672 · multiple benchmarks (average) | average score: +6.6% | Helped |
| Increase visual instruction data from 665K to 4M (ablation) | HaploVL with Llama-3-8B at 672 × 672 · GQA | score: 65.7 -> 65.5 | Hurt |
| Remove modal expansion Stage 1 and use next-token prediction directly (ablation) | HaploVL · multiple benchmarks | performance: 4.3% | Hurt |
| Add modal expansion Stage 1 with teacher-guided pre-decoder training (ablation) | HaploVL pre-decoder · ImageNet-1K | zero-shot accuracy: 0.1 -> 71.7 | Helped |
| Replace fixed-position mixed data with interleaved sequences (ablation) | HaploVL Stage 1 pre-training · multiple benchmarks (average) | average score: 59.0 -> 59.2 | Helped |
| Use MMC4 interleaved data instead of mix-v2 (ablation) | HaploVL Stage 1 pre-training at resolution 224 · multiple benchmarks (average) | average score: 59.2 -> 59.4 | Helped |
| Increase vision-teacher resolution from 224 to 336 (ablation) | HaploVL Stage 1 pre-training with mix-v2 · multiple benchmarks (average) | average score: +1.5% | Helped |
| Add pure text data to Stage 1 training (ablation) | HaploVL Stage 1 pre-training at resolution 336 · multiple benchmarks (average) | average score: 61.2 -> 61.3 | Helped |
Surprising findings
- Pure next-token prediction without the distillation stage destroys zero-shot ImageNet accuracy (0.1% vs 76.6% teacher), so the pre-decoder forgets its vision knowledge.
Limitations
- Performance still falls short of LLaVA-OneVision, attributed to lower input resolution (up to 2304 tokens) and limited context length (6144).
- Evaluation is restricted to image benchmarks; video and multi-image results are qualitative or limited to one table.
Experimental setup
Datasets: LLaVA-1.5 665K instruction data LLaVA 558K caption data MMC4 Dolphin Alpaca ShareGPT CC3M COCO LLaVA-OneVision 4M subset ImageNet-1K
Benchmarks: SEED-Bench POPE AI2D RealWorldQA MMMU MMBench MMStar VQAv2 GQA ScienceQA-IMG MMVP ImageNet-1K
Metrics: accuracy
Baselines: LLaVA-1.5-7B LLaVA-1.6 EVE-7B Emu3-8B Chameleon-30B Fuyu-8B LLaVA-OneVision
Builds on: LLaVA-1.5 Fuyu EVE CLIP Llama-3
Closest papers by meaning
- MoDA: Modulation Adapter for Fine-Grained Visual Understanding in Instructional MLLMs (ICML 2026 · similarity 0.89)
- L-Man: A Large Multi-modal Model Unifying Human-centric Tasks (AAAI 2025 · similarity 0.89)
- Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs (NeurIPS 2026 · similarity 0.89)
- ModaVerse: Efficiently Transforming Modalities with LLMs (CVPR 2024 · similarity 0.88)
- Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs (CVPR 2024 · similarity 0.88)
- IAA: Inner-Adaptor Architecture Empowers Frozen Large Language Model with Multimodal Capabilities (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.