Research map / Vision-language and multimodal / Multimodal LLMs

Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactions

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Align-TI, a white-box MLLM distillation framework that adds instruction-aware visual token alignment (IVA) and token-to-token transition probability alignment (TPA) to standard next-token KD, distilling Qwen2/Qwen3 teachers into 0.5B-2B students.

Problem

Existing MLLM knowledge distillation relies on static next-token alignment over off-policy ground-truth prefixes, ignoring vision-instruction token interactions and intra-response transition dynamics, which limits transfer of understanding and generation capabilities to compact students.

Approach

Align the student with the teacher on (1) instruction-to-vision attention-weighted visual tokens from the layer with maximal Instruction-Relevance Score, and (2) one-step token transition distributions conditioned on sampled student tokens, computed in parallel via a ribbon attention mask.

Key results

• Align-TI with Qwen2-1.5B reaches 72.2 average and Align-TI with Qwen3-1.7B reaches 73.6 average, versus 68.8 for LLaVA-1.5-7B.
• Align-TI with Qwen3-0.6B achieves 69.7 average on the ~1B setting, the best among compact models reported.
• In the ablation with Qwen2-0.5B, Align-TI (IVA+TPA) reaches 66.7 average vs 64.3 for the SFT-style baseline, with TPA contributing +2.1 and IVA +0.8; training time rises 1.4x over Vanilla KD (509 vs 355 H).

Surprising findings

• Greedy-based TPA sampling underperforms nucleus sampling at low sampled-token counts d.
• Vanilla KD gives only 0.2 average gain over SFT on MobileLLaMA-1.4B and drops 1.5 on MME.

Limitations

• Gains of IVA are modest (about 0.8 average alone) and TPA carries most of the training overhead.
• Benefits of larger teachers show diminishing returns beyond 4B for the student capacity considered.

Experimental setup

Datasets: ShareGPT4V-PT LLaVA-mix-665k LLaVA1.5-558k VSR SQA TextVQA VIGC IConQA Visual Dialog COCO SBU

Benchmarks: GQA SQA TextVQA POPE MME MMBench

Metrics: accuracy average score training time memory %ExAccErr

Baselines: Vanilla KD SFT FKL MiniLLM JSD GKD LLaVA-MoD LLaVA-KD Align-KD MoVE-KD LLaVA-1.5-7B MobileVLM V2

Builds on: LLaVA MiniLLM GKD LLaVA-MoD LLaVA-KD

Closest papers by meaning

• Rethinking Visual Token Reduction in LVLMs Under Cross-Modal Misalignment (AAAI 2026 · similarity 0.90)
• TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs (NeurIPS 2026 · similarity 0.90)
• Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings (NeurIPS 2025 · similarity 0.90)
• Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models (NeurIPS 2025 · similarity 0.90)
• Task-Related Token Compression in Multimodal Large Language Models from an Explainability Perspective (ICLR 2026 · similarity 0.90)
• Towards Multimodal Large Language Models with Both Training and Inference Efficiency (ICML 2026 · similarity 0.90)