Research map / Vision-language and multimodal / Multimodal LLMs

Towards Long-window Anchoring in Vision-Language Model Distillation

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LAid, a head-level distillation from Qwen2.5-VL-32B into smaller Qwen2.5-VL students (7B, 3B) that transfers long-range positional attention, extending effective visual-haystack window size.

Problem

Small vision-language models lose accuracy quickly as the number of input images (window length) grows, while large sibling models keep long-window performance; post-training methods to close this gap are under-explored for VLMs.

Approach

Treat large VLMs as long-window anchors and distill head-level query/key representations from multiple teacher heads into each student head via learned weights (plus a learnable RoPE response gain), so the student inherits low-frequency long-range positional attention, combined with standard KL and SFT losses.

Key results

• LAid-distilled Qwen2.5-VL-7B reaches 63.37% accuracy at 100 images versus 51.08% for the base 7B model and 62.56% for the 32B teacher.
• On the 3B student, LAid gives +16.4% average relative gain on long windows versus base, while YaRN gives -1.9% and SelfExtend -23.26%.
• Standard SFT(LoRA) gives +35.92% short-window gain on 7B but only +3.6% on long windows.

Surprising findings

• Direct application of YaRN and SelfExtend degrades long-window accuracy on VLMs.
• SFT improves short-window accuracy strongly but yields little long-window benefit.

Limitations

• Focuses on attention components and does not transfer feed-forward networks.
• Distillation adds training overhead.
• Evaluated on a single benchmark (Visual HayStacks) with binary yes/no questions.

Experimental setup

Datasets: Visual HayStacks (VHs) COCO

Benchmarks: Visual HayStacks

Metrics: accuracy relative gain vs base across haystack sizes

Baselines: YaRN SelfExtend SFT (LoRA) standard KD

Builds on: RoPE YaRN SelfExtend LoRA Visual Haystacks

Closest papers by meaning

• HoPE: Hybrid of Position Embedding for Long Context Vision-Language Models (NeurIPS 2025 · similarity 0.92)
• Large Vision-Language Models Get Lost in Attention (ICML 2026 · similarity 0.90)
• Revisiting Multimodal Positional Encoding in Vision–Language Models (ICLR 2026 · similarity 0.90)
• Rope to Nope and Back Again: A New Hybrid Attention Strategy (NeurIPS 2025 · similarity 0.89)
• Mitigating Mask Prior Drift and Positional Attention Collapse in Large Diffusion Vision-Language Models (ICML 2026 · similarity 0.89)
• In-Context Learning Can Help Vision Language Models Overcome Training Prior (NeurIPS 2026 · similarity 0.89)