Research map / Vision-language and multimodal / Cross-modal learning

Align-KD: Distilling Cross-Modal Alignment Knowledge for Mobile Vision-Language Large Model Enhancement

CVPR 2025 · Vision-language and multimodal › Cross-modal learning · approach: Contrastive vision-language alignment · 11 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Align-KD distills cross-modal alignment knowledge from a 7B MobileVLM V2 teacher into a 1.7B student by matching first-layer text-to-vision attention and text-focused vision tokens, plus reverse KL on outputs.

Problem

Compressing vision-language models for mobile devices causes performance drops, and existing LLM/VLM distillation methods ignore cross-modal alignment knowledge.

Approach

Distill cross-modal alignment specifically: (1) mimic only the teacher's first-layer text-query-vision attention via MSE, and (2) inject teacher vision-token embeddings into the projector output, weighted toward the top-K tokens most attended by text, alongside reverse-KLD output distillation.

Key results

• On the Short subset, Align-KD raises MobileVLM V2 1.7B average from 62.4 to 64.4 (+2.0) across 6 benchmarks.
• On the Long subset, Align-KD raises average from 63.7 to 65.1, with SQA-IMG up 3.2 and MMBench up 1.6 for the Short-subset setting.
• Distilling only first-layer text-query-vision attention gives 63.6 average vs 62.4 baseline in ablation, and adding vision-token focus loss yields 64.4; distilling vision-query-vision attention drops average to 43.7.

Measured effects reported in the paper

Surprising findings

• Distilling the last layer's full attention in addition to Align-KD reduces the average from 64.4 to 64.1 (reported as a performance drop relative to Align-KD alone).
• Distilling vision-query-vision attention causes extreme performance collapse (average 43.7), while text-query-text distillation is only slightly below the proposed design.

Limitations

• Experiments are limited to one student (MobileVLM V2 1.7B) and one teacher (7B) from a single family.
• Training cost increases substantially (296 vs 176 GPU hours on Short; 676 vs 228 on Long), and the teacher must be run during training.
• Gains are reported on six benchmarks with a single training run and no variance estimates.

Experimental setup

Datasets: ShareGPT4V-PT COCO SBU Visual Dialog ShareGPT4V SQA IConQA TextVQA VSR VIGC

Benchmarks: GQA SQA TextVQA MME MMBench POPE

Metrics: accuracy average score across 6 benchmarks

Baselines: MobileVLM V2 1.7B without distillation +RKLD +A1,v-v +A1,t-t +A1,all +A_last,all

Builds on: MiniLLM MobileVLM V2 LLaVA-MoD Sun et al. layer analysis of LLaMA

Closest papers by meaning

• Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactions (ICML 2026 · similarity 0.89)
• Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models (NeurIPS 2025 · similarity 0.89)
• CovMatch: Cross-Covariance Guided Multimodal Dataset Distillation with Trainable Text Encoder (NeurIPS 2025 · similarity 0.88)
• LVLM-Aided Alignment of Task-Specific Vision Models (CVPR 2026 · similarity 0.88)
• Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs (ICML 2026 · similarity 0.88)
• DeepAlign: Mitigating Modality Conflict through Modality-Specific Alignment (CVPR 2026 · similarity 0.88)