Research map / Vision-language and multimodal / Multimodal LLMs
Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VisPer-LM, which distills predictive visual embeddings from expert depth, segmentation, and generation encoders into intermediate LLM layers during MLLM pretraining (with special tokens), keeping a single encoder at inference, and reports gains on CV-Bench depth/distance/relation tasks.
Problem
Standard MLLMs trained only with next-token prediction on vision encoder features under-represent fine-grained visual perception (depth, spatial relations), which hurts spatial reasoning for embodied AI and robotics, while multi-encoder approaches add inference latency and data/compute cost.
Approach
Instead of feeding auxiliary expert encoder features into the LLM, distill their representations into intermediate LLM hidden states during pretraining via a Perceiver-resampler embedding predictor per target layer, combining smooth-L1 and InfoNCE losses with the NTP loss, plus learnable task tokens; only the base encoder is used at inference.
Key results
- VisPer-LM (Llama3-8b, CLIP-ConvNeXT-XXL) reaches 8.7% gain over LLaVA-1.5 on CV-Bench Depth and up to 2.5% average gain across benchmarks.
- On CV-Bench with Llama3-8b and CLIP-ViT-L, VisPer-LM averages 57.6 vs 55.1 for LLaVA-1.5, and outperforms multi-encoder token concat baseline (56.7) on average.
- Single-encoder inference throughput is maintained (9.86 vs 9.92 samples/sec) while the multi-encoder baseline drops to 5.32 samples/sec.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add visual embedding distillation during PT | LLaVA-1.5 with Llama3-8b and CLIP-ConvNeXT-XXL · CV-Bench Depth (3D) | accuracy: 62.8 -> 71.5 | Helped |
| Add visual embedding distillation during PT | LLaVA-1.5 with Llama3-8b and CLIP-ConvNeXT-XXL · CV-Bench Relation (2D) | accuracy: 69.5 -> 66.8 | Hurt |
| Add visual embedding distillation during PT | LLaVA-1.5 with Llama3-8b and CLIP-ConvNeXT-XXL · CV-Bench Distance (3D) | accuracy: 49.8 -> 52.8 | Helped |
| Add visual embedding distillation during PT | LLaVA-1.5 with CLIP-ViT-L and Llama3-8b · CV-Bench | accuracy | Mixed |
| Use embedding losses during IFT as well as PT (ablation) | VisPer-LM · CV-Bench 3D | accuracy: 64.2 -> 58.3 | Hurt |
| Add learnable task tokens without embedding loss (ablation) | Single-encoder baseline · MMStar | accuracy: 38.8 -> 40.1 | Helped |
| Use 8 special task tokens (ablation) | VisPer-LM · CV-Bench 3D | accuracy: 61.0 -> 64.2 | Helped |
| Use only the segmentation embedding loss (ablation) | VisPer-LM · CV-Bench 3D | accuracy: 61.0 -> 57.6 | Hurt |
| Use all three depth, segmentation, and generation embedding losses (ablation) | VisPer-LM · MMStar | accuracy: 38.8 -> 39.5 | Helped |
| Add contrastive loss to Smooth-L1 embedding loss (ablation) | VisPer-LM · CV-Bench 3D | accuracy: 62.3 -> 64.2 | Helped |
| Freeze special task tokens during IFT (ablation) | VisPer-LM · CV-Bench 3D | accuracy: 56.1 -> 64.2 | Helped |
| Add visual pretraining on ALLaVA-Caption-663K (ablation) | VisPer-LM · CV-Bench overall | accuracy: 63.3 -> 64.6 | Helped |
Surprising findings
- Multi-encoder baselines beat VisPer-LM on the CV-Bench Depth task, and the Distance probe correlation with probe scores is negative.
- Adding the embedding losses during IFT hurts performance (avg 53.0 vs 55.1 with PT only).
Limitations
- Does not improve general reasoning beyond the baseline; focus is perception, and multi-encoder baseline remains better on Depth.
- Only evaluated with LLaVA-1.5 setup and up to 8B LLMs; not scaled to 70B-scale models.
- Limited to image modality; video/motion not explored.
Experimental setup
Datasets: LLaVA-558K LLaVA-665K ALLaVA-Caption-663K COCO-train2017 COCO-val2017 CV-Bench
Benchmarks: CV-Bench MMStar RWQA OK-VQA POPE GQA MMMU VizWiz BLINK DA-2K
Metrics: accuracy cosine similarity (probing) throughput (samples/sec) FID mIoU
Baselines: LLaVA-1.5 Cambrian-1 RADIO-ViT-L LLaVA-1.5 multi-encoder feature concat multi-encoder token concat EVE-7B Chameleon-7B
Builds on: LLaVA-1.5 I-JEPA REPA Perceiver Resampler
Closest papers by meaning
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs (NeurIPS 2024 · similarity 0.90)
- Learning to See Before Seeing: Demystifying LLM Visual Priors from Language Pre-training (ICLR 2026 · similarity 0.90)
- VCoder: Versatile Vision Encoders for Multimodal Large Language Models (CVPR 2024 · similarity 0.90)
- PerceptionGPT: Effectively Fusing Visual Perception into LLM (CVPR 2024 · similarity 0.90)
- UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm (CVPR 2026 · similarity 0.90)
- Perception Tokens Enhance Visual Reasoning in Multimodal Language Models (CVPR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.