Research map / Vision-language and multimodal / Multimodal LLMs

Gated Relational Alignment via Confidence-based Distillation for Efficient VLMs

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes GRACE, a QAT-plus-distillation framework for compressing VLMs (LLaVA-1.5 and Qwen2-VL) that combines confidence-gated decoupled KD, relational CKA alignment of visual tokens, and an adaptive IB controller, reporting INT4 models that exceed BF16 baselines.

Problem

Post-training quantization causes significant accuracy loss for VLMs at INT4, and quantization-aware training for VLMs is underexplored; naive KD combined with QAT treats all teacher predictions as equally reliable.

Approach

Unifies KD and QAT under an Information Bottleneck view: quantization limits student capacity, teacher entropy gates distillation (confidence-weighted decoupled KD), CKA over intra-sample visual-token Gram matrices transfers relational structure, and a dual-ascent controller sets the distillation weight online.

Key results

• 4-bit GRACE on LLaVA-1.5-7B averages 67.2 vs 66.5 BF16 baseline and 65.9 for QAT, with ScienceQA 70.1 vs 66.8.
• On Qwen2-VL-2B, 4-bit GRACE averages 68.0 vs 64.0 BF16 baseline and outperforms SPEED-Q (62.3).
• With TinyChat INT4 kernels, the 7B INT4 model reaches 3.03x throughput (106.68 vs 35.26 tokens/s) and 2.18x lower peak memory versus FP16.

Surprising findings

• The INT4 student surpasses its BF16 baseline on several benchmarks, and the Qwen3-VL-2B student slightly exceeds its 8B teacher on average (76.7 vs 76.3).
• Naive PTQ (GPTQ/AWQ) applied after GRACE distillation still loses 3.3-3.7 points, while joint QAT+KD preserves nearly all distillation gains.

Limitations

• Only weight-only quantization is studied; activation quantization is left unexplored.
• Requires a stronger teacher model during training, increasing offline training cost.
• Experiments are limited to image-text VLMs, not video or audio language models.

Experimental setup

Datasets: ShareGPT4V

Benchmarks: VQAv2 GQA TextVQA VizWiz POPE ScienceQA MME MMBench SEED-Bench MMMU MMStar HallusionBench

Metrics: accuracy average accuracy throughput (tokens/s) latency (ms/token) peak GPU memory parameter size

Baselines: MoVE-KD HAWAII RTN AWQ GPTQ MBQ SPEED-Q QAT LLaVA-1.5-13B teacher

Builds on: decoupled knowledge distillation LSQ centered kernel alignment relational knowledge distillation TinyChat

Closest papers by meaning

• Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients (CVPR 2026 · similarity 0.89)
• Quant Experts: Token-aware Adaptive Error Reconstruction with Mixture of Experts for Large Vision-Language Models Quantization (CVPR 2026 · similarity 0.89)
• Beyond Next-Token Alignment: Distilling Multimodal Large Language Models via Token Interactions (ICML 2026 · similarity 0.88)
• Q-VLM: Post-training Quantization for Large Vision-Language Models (NeurIPS 2024 · similarity 0.88)
• Deep Pre-Alignment for VLMs (ICML 2026 · similarity 0.88)
• VLM-PTQ: Efficient Post-Training Quantization for Large Vision-Language Models (CVPR 2026 · similarity 0.88)