Research map / Label-efficient and robust learning / Knowledge distillation
Compressing Visual-Linguistic Model via Knowledge Distillation
Zhiyuan Fang, Jianfeng Wang, Xiaowei Hu, Lijuan Wang, Yezhou Yang, Zicheng Liu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add hidden embedding (NCE) distillation (ablation) | VLP small VL model · COCO Captioning | CIDEr: 110.6 -> 114.4 | Improved |
| add attention distribution distillation (ablation) | VLP small VL model · COCO Captioning | CIDEr: 110.6 -> 111.8 | Improved |
| add both attention and hidden distillation (ablation) | VLP small VL model · COCO Captioning | CIDEr: 110.6 -> 115.6 | Improved |
| pre-training distillation with all losses (ablation) | DistillVLM · COCO Captioning | CIDEr: 115.7 -> 120.1 | Improved |
| add fine-tuning classification distillation (ablation) | DistillVLM · COCO Captioning | CIDEr: 115.7 -> 117.1 | Improved |
| add fine-tuning classification distillation (ablation) | DistillVLM · VQA 2.0 test-std | accuracy: 69.0 -> 69.2 | No clear effect |
| add attention or hidden distillation at fine-tuning on top of classification distillation (ablation) | DistillVLM · COCO Captioning | CIDEr: 116.3 -> 116.1 | No clear effect |
| use last-layer instead of layer-to-layer MSE distillation (ablation) | DistillVLM · COCO Captioning | CIDEr: 114.8 -> 115.3 | Improved |
| replace MSE with NCE contrastive loss on last layer (ablation) | DistillVLM · COCO Captioning | CIDEr: 115.3 -> 115.6 | Improved |
| use token-wise embeddings instead of mean-pooled embedding for NCE (ablation) | DistillVLM · COCO Captioning | CIDEr: 115.4 -> 115.6 | Improved |
| increase NCE queue size from 1 to 4,096 negatives (ablation) | DistillVLM · COCO Captioning | CIDEr: 112.5 -> 115.4 | Improved |
| align visual tokens using student proposals (teacher not retrained) (ablation) | DistillVLM · COCO Captioning | CIDEr: 112.3 -> 115.0 | Improved |
Closest papers by meaning
- MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders (CVPR 2025 · similarity 0.90)
- Efficient Large Multi-modal Models via Visual Context Compression (NeurIPS 2024 · similarity 0.90)
- Probing Visual Language Priors in VLMs (ICML 2025 · similarity 0.89)
- No Head Left Behind – Multi-Head Alignment Distillation for Transformers (AAAI 2024 · similarity 0.89)
- Rethinking Token Reduction for Large Vision-Language Models (CVPR 2026 · similarity 0.89)
- MoDA: Modulation Adapter for Fine-Grained Visual Understanding in Instructional MLLMs (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
