Research map / Vision-language and multimodal / Human and embodied modeling
VL2Lite: Task-Specific Knowledge Distillation from Large Vision-Language Models to Lightweight Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VL2Lite, a one-step distillation framework that transfers visual (relational) and linguistic (text-similarity) knowledge from a frozen OpenCLIP ConvNeXt-XXL VLM into lightweight classifiers during task-specific training.
Problem
Lightweight neural networks lose accuracy relative to large models, and traditional knowledge distillation requires a costly two-stage process (pre-training a teacher on the target task then distilling), with risk of transferring teacher biases.
Approach
Keep a large VLM frozen and jointly train the student with cross-entropy plus a relational (pairwise-distance, smooth L1) visual distillation loss and a text-similarity KL loss, linked to the VLM through trainable condensation MLP layers that project VLM features to the student's dimension; no separate teacher fine-tuning is needed.
Key results
- VL2Lite improves ResNet-18 top-1 accuracy on CUB-200-2011 from 64.95% to 71.38% (+6.43%) and Stanford Cars by +7.06%.
- Across six backbones (ResNet-18/34, EfficientNet-B0, MobileNet-V2, ShuffleNet-V2, Swin-Tiny), VL2Lite improves accuracy on every dataset, with gains up to about 8.8%.
- With 30% of training data on CUB-200, ResNet-18 accuracy rises from 39.61% to 50.32% (+10.71%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add VL2Lite visual and linguistic distillation to ResNet-18 | ResNet-18 · CUB-200 | accuracy: 64.95% -> 71.38% (+6.43%) | Helped |
| add VL2Lite visual and linguistic distillation to ResNet-18 | ResNet-18 · FGVC | accuracy: 50.98% -> 55.87% (+4.89%) | Helped |
| add VL2Lite visual and linguistic distillation to ResNet-18 | ResNet-18 · NABirds | accuracy: 57.01% -> 63.26% (+6.25%) | Helped |
| add VL2Lite visual and linguistic distillation to ResNet-18 | ResNet-18 · DTD | accuracy: 67.25% -> 71.78% (+4.53%) | Helped |
| add VL2Lite visual and linguistic distillation to ResNet-18 | ResNet-18 · Oxford Pet | accuracy: 85.82% -> 88.72% (+2.90%) | Helped |
| add VL2Lite visual and linguistic distillation to ResNet-18 | ResNet-18 · Sf Dogs | accuracy: 67.80% -> 72.40% (+4.60%) | Helped |
| add VL2Lite visual and linguistic distillation to ResNet-18 | ResNet-18 · Sf Cars | accuracy: 70.03% -> 77.09% (+7.06%) | Helped |
| add VL2Lite visual and linguistic distillation to ResNet-18 | ResNet-18 · Ct-101 | accuracy: 90.45% -> 92.33% (+1.88%) | Helped |
| add VL2Lite visual and linguistic distillation to ResNet-18 | ResNet-18 · Ct-256 | accuracy: 77.58% -> 79.28% (+1.70%) | Helped |
| add visual distillation loss (ablation) | ResNet-18 with task classification loss · CUB-200, FGVC, DTD, Oxford Pet, Sf Dogs, Sf Cars | accuracy: 64.95% -> 68.50%; 50.98% -> 53.31%; 67.25% -> 68.46%; 85.82% -> 87.32%; 67.80% -> 68.85%; 70.03% -> 71.86% | Helped |
| add linguistic distillation loss (ablation) | ResNet-18 with task classification loss · CUB-200, FGVC, DTD, Oxford Pet, Sf Dogs, Sf Cars | accuracy: 64.95% -> 71.03%; 50.98% -> 54.84%; 67.25% -> 69.42%; 85.82% -> 87.98%; 67.80% -> 71.86%; 70.03% -> 76.68% | Helped |
| combine task classification, visual distillation, and linguistic distillation losses (ablation) | ResNet-18 · CUB-200, FGVC, DTD, Oxford Pet, Sf Dogs, Sf Cars | accuracy: 68.50% -> 71.38%; 53.31% -> 55.87%; 68.46% -> 71.78%; 87.32% -> 88.72%; 68.85% -> 72.40%; 71.86% -> 77.09% | Helped |
Surprising findings
- Linguistic (text-similarity) distillation gives larger gains than the visual relational distillation loss, per the ablation.
- Cross-modal image-text distillation (as in RISE) performed worse than within-modality relational distillation in the reported comparisons.
Limitations
- Training requires running a large frozen VLM (ConvNeXt-XXLarge) forward pass, which adds training cost.
- Evaluation is limited to image classification on nine datasets with single-run accuracy reported and no variance estimates.
Experimental setup
Datasets: CUB-200-2011 FGVC-Aircraft NABirds DTD Oxford-IIIT Pet Stanford Dogs Stanford Cars Caltech-101 Caltech-256
Benchmarks: CUB-200-2011 FGVC-Aircraft NABirds DTD Oxford-IIIT Pet Stanford Dogs Stanford Cars Caltech-101 Caltech-256
Metrics: top-1 accuracy
Baselines: KD (T: R-152) RKD (T: R-152) KD (T: CLIP) RKD (T: CLIP) RISE BorLan w/o KD
Builds on: Relational Knowledge Distillation CLIP RISE
Closest papers by meaning
- MoVE-KD: Knowledge Distillation for VLMs with Mixture of Visual Encoders (CVPR 2025 · similarity 0.88)
- Leveraging Vision-Language Models for Improving Domain Generalization in Image Classification (CVPR 2024 · similarity 0.88)
- ScaleKD: Strong Vision Transformers Could Be Excellent Teachers (NeurIPS 2024 · similarity 0.87)
- Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation (ICLR 2026 · similarity 0.87)
- Vision‑Language‑Vision Auto‑Encoder: Scalable Knowledge Distillation from Diffusion Models (NeurIPS 2025 · similarity 0.87)
- Towards General Continuous Memory for Vision-Language Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.