Research map / Vision-language and multimodal / Human and embodied modeling
No Head Left Behind – Multi-Head Alignment Distillation for Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Attention Map Alignment Distillation (AMAD), which soft-aligns teacher and student attention heads via cosine similarity so attention-map distillation works when teacher and student have different head counts, and applies it to distill VL-T5 and BLIP.
Problem
Conventional attention-map distillation requires the teacher and student to have the same number of attention heads and a one-to-one head correspondence, which fails for vision-language Transformers with differing head counts.
Approach
Soft-align teacher and student attention heads using softmax-normalized cosine similarities so each teacher head distills into all student heads in proportion to similarity, making the loss head-count agnostic and order-free (operates like cross-attention).
Key results
- Distilled VL-T5 small without VL pre-training (AMAD) reaches 68.06 VQA Karpathy accuracy with VL pre-training and 67.79 without, both beating the GT-fine-tuned VL-T5 small baseline (66.72 with VLP).
- On COCO captioning, AMAD raises BLIP base student CIDEr from 65.1 to 95.0 when trained without pre-training.
- On Multi30K, the AMAD-distilled VL-T5 small improves BLEU@4 by +1.98, +1.85 and +2.62 over the ground-truth-trained student across test splits.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add AMAD attention-map distillation to logits distillation | VL-T5 small · COCO Captioning Karpathy test | CIDEr: 108.2 -> 114.4 | Helped |
| add AMAD attention-map distillation to logits distillation | VL-T5 small · VQA-2.0 test-std | accuracy: 69.28 -> 70.47 | Helped |
| fine-tune with distillation from a pre-trained Teacher despite no Student VL pre-training | VL-T5 small · COCO Captioning Karpathy test | CIDEr: 101.4 -> 112.9 | Helped |
| fine-tune with distillation from a pre-trained Teacher despite no Student VL pre-training | VL-T5 small · VQA-2.0 test-dev | accuracy: 58.47 -> 70.06 | Helped |
| fine-tune with distillation from BLIP large without Student pre-training | BLIP base · COCO Captioning Karpathy test | CIDEr: 65.1 -> 95.0 | Helped |
| add AMAD to logits distillation (ablation) | VL-T5 small · COCO Captioning Karpathy test | CIDEr: 111.8 -> 114.4 | Helped |
| add AMAD to logits distillation (ablation) | VL-T5 small · VQA-2.0 test-std | accuracy: 70.19 -> 70.47 | Helped |
| add AMAD to logits distillation without Student VL pre-training (ablation) | VL-T5 small · VQA-2.0 test-dev | accuracy: 68.71 -> 70.06 | Helped |
| replace AMAD with one-to-one attention-map distillation (ablation) | VL-T5 small with logits distillation · VQA-2.0 Karpathy test | accuracy: 67.74 -> 67.73 | No clear effect |
| use AMAD Variant 1 (MSE) instead of logits distillation alone (ablation) | VL-T5 small · VQA-2.0 Karpathy test | accuracy: 67.74 -> 67.96 | Helped |
| use AMAD Variant 2 (KL divergence) instead of logits distillation alone (ablation) | VL-T5 small · VQA-2.0 Karpathy test | accuracy: 67.74 -> 68.06 | Helped |
| use AMAD Variant 3 (parameterized projection) instead of logits distillation alone (ablation) | VL-T5 small · VQA-2.0 Karpathy test | accuracy: 67.74 -> 68.02 | Helped |
Surprising findings
- Fine-tuning distillation without any VL pre-training lets the VL-T5 small student outperform the VL pre-trained, ground-truth fine-tuned student.
- Naive one-to-one attention distillation (LKD-ATT) did not improve over logits distillation alone and slightly hurt results.
Limitations
- Distilled BLIP students without VL pre-training still perform far below the teacher, showing uni-modal pre-training remains necessary.
- VL-T5 small was pre-trained by the authors because no released checkpoint exists, and the teacher was reproduced, so comparisons depend on reproduced baselines.
- Distillation is applied only during downstream fine-tuning, not pre-training.
Experimental setup
Datasets: VQA-2.0 COCO Captioning (Karpathy split) Multi30K MS COCO Visual Genome GQA Visual7W
Benchmarks: VQA-2.0 COCO Captioning Multi30K
Metrics: VQA accuracy BLEU@4 CIDEr METEOR SPICE SacreBLEU
Baselines: one-to-one attention distillation (LKD-ATT) logits distillation (LKD) RKD-D DistillVLM MiniVLM MAD
Builds on: MiniLM VL-T5 BLIP DistillVLM Hinton knowledge distillation
Closest papers by meaning
- Towards Long-window Anchoring in Vision-Language Model Distillation (AAAI 2026 · similarity 0.89)
- Attention Transfer Is Not Universally Effective for Vision Transformers (NeurIPS 2026 · similarity 0.88)
- Memorization Capacity of Multi-Head Attention in Transformers (ICLR 2024 · similarity 0.88)
- Attention Alignment Between Humans and Vision-Language Models (NeurIPS 2026 · similarity 0.88)
- Head Pursuit: Probing Attention Specialization in Multimodal Transformers (NeurIPS 2025 · similarity 0.88)
- Mind the Heads: Topological Representation Alignment for Multimodal LLMs (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.