Research map / Vision-language and multimodal / Multimodal LLMs
VLsI: Verbalized Layers-to-Interactions from Large to Small Vision Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VLsI, a layer-wise natural-language distillation method that uses verbalizers (verb-FFN plus the shared language head) on intermediate layers to transfer knowledge from a large Qwen2-VL (72B) to smaller 2B/7B VLMs, with adaptive layer matching and an SFT stage.
Problem
Small vision-language models lag behind large open and closed VLMs on challenging benchmarks, and standard distillation mostly imitates final-layer outputs, which is unstable and ignores the layer-count gap between large and small models.
Approach
Attach verbalizers (a lightweight verb-FFN followed by the backbone's language head) to intermediate layers so each layer's features map to vocabulary distributions; align small-model layers to large-model layers via KL divergence with order-preserving, softmax-sampled layer matching, then run SFT.
Key results
- VLsI-7B reaches 75.8 on MM-Vet and 69.3 on MMMU, versus 54.1 MMMU for Qwen2-VL-7B and 64.3 for Qwen2-VL-72B reported in the ablation table.
- VLsI-2B scores 68.4 on MathVista and 64.8 on MM-Vet, outperforming the Qwen2-VL-2B baseline (43.0 and 49.5).
- Ablation shows KL divergence in both intermediate and last-layer interaction (KLD/KLD) gives the best results; removing the language head verbalizer drops MM-Vet to 63.3-66.5 range for alternatives.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use VLsI layer-wise distillation instead of the Qwen2-VL baseline | Qwen2-VL-7B · MM-Vet | score: 62.0 -> 75.2 | Helped |
| Use VLsI layer-wise distillation instead of the Qwen2-VL baseline | Qwen2-VL-2B · MM-Vet | score: 49.5 -> 64.8 | Helped |
| Add SFT after the interaction step (ablation) | VLsI-2B · MM-Vet | score: 47.9 -> 64.8 | Helped |
| Add SFT after the interaction step (ablation) | VLsI-7B · MM-Vet | score: 60.5 -> 75.8 | Helped |
| Use beam decoding (N=5) without SFT instead of greedy decoding (ablation) | VLsI-2B · MM-Vet | score: 47.9 -> 58.1 | Helped |
| Use KLD for both intermediate and last-layer operations instead of CE for both (ablation) | VLsI-7B · MM-Vet | score: 63.2 -> 75.8 | Helped |
| Add KLD for last-layer operations while retaining CE intermediate operations (ablation) | VLsI-7B · MM-Vet | score: 64.5 -> 67.2 | Helped |
| Add the verbalizer language head for verbalization (ablation) | VLsI-7B · MM-Vet | score: 66.5 -> 75.8 | Helped |
| Use multinomial sampling instead of random layer selection (ablation) | VLsI-7B matching strategy · MMB | score: 77.0 -> 82.0 | Helped |
| Add search range to multinomial sampling (ablation) | VLsI-7B matching strategy · MM-Vet | score: 68.5 -> 69.8 | Helped |
| Add order preservation to the matching strategy (ablation) | VLsI-7B matching strategy · MM-Vet | score: 69.8 -> 75.2 | Helped |
| Add adaptive temperature to the matching strategy (ablation) | VLsI-7B matching strategy · MMMU | score: 68.5 -> 69.3 | Helped |
Surprising findings
- Without the SFT step, the interaction-only VLsI-2B/7B underperform the Qwen2-VL baselines under greedy decoding, but beam search (N=5) substantially improves them, suggesting the interaction step expands the probability space over the language head.
Limitations
- Teacher and student must share the same tokenizer and token index order.
- Relies on large backbone VLM (72B) for distillation, so training requires access to a large model.
- Verbalization uses greedy decoding; results without SFT are weak under greedy decoding.
Experimental setup
Datasets: ShareGPT4o-Images ShareGPT4V ALLaVA-VFLAN/Text MiniGemini LLaVA-HD mPLUG-DocOwl1.5 GLLaVA MathVision MathInstruct MathPlus
Benchmarks: QBench AI2D ChartQA POPE HallusionBench MME MathVista MMBench MMBench-CN MM-Vet MMMU MM-Vet-v2
Metrics: accuracy
Baselines: Qwen2-VL-2B Qwen2-VL-7B TroL-7B Phantom-7B InternVL2-8B LLaVA-OneVision-7B GPT-4V Gemini-1.5-Pro Claude-3.5-Sonnet GPT-4o
Builds on: Qwen2-VL TroL Phantom speculative decoding LLaVA-OneVision
Closest papers by meaning
- EfficientVLA: Training-Free Acceleration and Compression for Vision-Language-Action Models (NeurIPS 2025 · similarity 0.90)
- What matters when building vision-language models? (NeurIPS 2024 · similarity 0.90)
- Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models (ICML 2024 · similarity 0.90)
- From Pixels to Words -- Towards Native Vision-Language Primitives at Scale (ICLR 2026 · similarity 0.89)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.89)
- ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.