Research map / Vision-language and multimodal / Multimodal LLMs
ToVE: Efficient Vision-Language Learning via Knowledge Transfer from Vision Experts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ToVE, which transfers knowledge from a hub of pre-trained vision experts into CLIP vision tokens via a token-aware gating network and residual knowledge transfer, then merges expert knowledge into a single CLIP encoder, achieving competitive VL performance with ~3M pre-training pairs.
Problem
Efficient vision-language learning without massive datasets or huge models; small VLMs lose fine-grained and spatial perception, and CLIP's visual perception is limited, causing spatial reasoning errors and object hallucination.
Approach
Route knowledge from multiple frozen pre-trained vision experts (depth, normal, edge, DINO, EVA) to each vision token via a token-aware softmax gate, add it residually (scaled by lambda) to the CLIP tokens, allow low-contribution experts to be detached at inference, and distill the expert-enhanced tokens back into the CLIP encoder so expert inference is unnecessary.
Key results
- Zero-shot NoCaps CIDEr of 110.2 with 3M pairs, vs 107.9 for Prismer-Large (12.7M pairs) and 87.5 for Prismer-Base; ToVE-Lite reaches 104.1.
- VSR accuracy of 67.7 for ToVE (3M) vs 55.3 for reproduced Prismer-Z at 3M pairs and 63.2 for Prismer-Z at 12.7M.
- Fine-tuned NoCaps overall CIDEr of 112.5 and COCO CIDEr of 132.8 (ablation table) reported for base-size models trained on 3M pairs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| transfer knowledge from all five vision experts | CLIP-based ToVE · NoCaps, VQAv2, VSR, POPE-R and POPE-A | benchmark scores | Helped |
| replace expert inference with knowledge merging into CLIP | ToVE · NoCaps zero-shot | CIDEr: 110.2 vs 104.1 | Hurt |
| add DINO and Depth experts | LLaVA-1.5-7B · MME perception | score: 1434.5 to 1523.1 | Helped |
| add DINO and Depth experts | LLaVA-1.5-7B · MMStar Fine-grained | score: improvement of 3.6 points | Helped |
| add Depth as the sole transferred expert (ablation) | CLIP-based ToVE · NoCaps | CIDEr: 92.1 -> 98.2 | Helped |
| add Edge as the sole transferred expert (ablation) | CLIP-based ToVE · NoCaps | CIDEr: 92.1 -> 96.3 | Helped |
| add Normal as the sole transferred expert (ablation) | CLIP-based ToVE · NoCaps | CIDEr: 92.1 -> 98.7 | Helped |
| add DINO as the sole transferred expert (ablation) | CLIP-based ToVE · NoCaps | CIDEr: 92.1 -> 105.1 | Helped |
| add EVA as the sole transferred expert (ablation) | CLIP-based ToVE · NoCaps | CIDEr: 92.1 -> 109.1 | Helped |
| increase the residual knowledge-fusion coefficient beyond 0.1 (ablation) | ToVE · NoCaps | CIDEr: 110.2 to 35.9 | Hurt |
| replace residual addition with direct addition (ablation) | ToVE · NoCaps | CIDEr: 110.2 -> 26.8 | Hurt |
| replace residual addition with concatenation (ablation) | ToVE · NoCaps | CIDEr: 110.2 -> 108.2 | Hurt |
Surprising findings
- Removing all vision experts causes model collapse, with the model failing at normal language generation.
- Without load balancing, the gating network converges to a trivial solution routing mostly to the EVA expert.
Limitations
- Depends on availability and quality of pre-trained vision experts.
- Full ToVE with multiple experts is computationally intensive at inference (1.2B total parameters).
- Effectiveness on unseen domains may vary and may need fine-tuning.
Experimental setup
Datasets: COCO Visual Genome CC3M MIX-6 ScanNet BIPED LVD-142M Merged-38M VQAv2 LLaVA instruction data
Benchmarks: NoCaps COCO Caption VQAv2 VSR POPE MME MMStar
Metrics: CIDEr SPICE BLEU-4 accuracy F1-score
Baselines: Prismer-Z Prismer BLIP BLIP-2 SimVLM VinVL OSCAR MiniGPT-4 LLaVA InstructBLIP
Builds on: Prismer CLIP MoE load balancing loss LLaVA-1.5
Closest papers by meaning
- Hawaii: Hierarchical Visual Knowledge Transfer for Efficient Vision-Language Models (NeurIPS 2025 · similarity 0.90)
- MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders (CVPR 2025 · similarity 0.90)
- MoVA: Adapting Mixture of Vision Experts to Multimodal Context (NeurIPS 2024 · similarity 0.89)
- Unveiling Encoder-Free Vision-Language Models (NeurIPS 2024 · similarity 0.88)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.88)
- Synthesize Diagnose and Optimize: Towards Fine-Grained Vision-Language Understanding (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.