Research map / Vision-language and multimodal / Multimodal LLMs

MHA2MLA-VLM: Enabling DeepSeek’s Economical Multi-Head Latent Attention Across Vision-Language Models

AAAI 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Converts off-the-shelf VLMs (LLaVA-1.5, LLaVA-NeXT, Qwen2.5-VL) from MHA/GQA to DeepSeek's MLA using modality-adaptive partial-RoPE, modality-decoupled output-activation SVD, and two-stage PEFT, shrinking KV cache by up to ~94% with small accuracy loss.

Problem

KV cache growth in vision-language models with long multimodal contexts creates memory and bandwidth bottlenecks, and it is unclear how to adapt existing MHA/GQA VLMs to the MLA architecture without expensive pretraining.

Approach

Extends the MHA2MLA conversion to VLMs with (1) a KL-divergence-based, modality-aware selection of retained RoPE/M-RoPE subspaces, and (2) separate visual and text low-rank SVD projections that minimize output activation error (SVD-LLM V2 style) rather than weight error, combined with a two-stage parameter-efficient fine-tuning schedule.

Key results

• On Qwen2.5-VL-7B, GQA2MLA with d_kv=64 (94.64% KV cache reduction) averages 79.47 vs 80.75 for the fine-tuned GQA baseline (-1.28).
• On LLaVA-NeXT-8B, GQA2MLA with d_kv=128 (84.38% reduction) averages 70.23 vs 70.78 for the original model.
• At ~62.5% KV reduction on LLaVA-NeXT, MHA2MLA-style compression averages 68.75 versus 63.38 for H2O and 60.48 for TOVA; combining with Int4 quantization keeps scores near 70.

Surprising findings

• Cache pruning baselines (H2O, TOVA) degrade more than the learned low-rank conversion at comparable compression ratios.
• Joint-modality SVD yields a larger loss than split-modality SVD, with the layer-wise gap reaching 35.85% at the final layer of Qwen2.5-VL.

Limitations

• Accuracy degrades noticeably at the most aggressive compression (e.g., LLaVA-NeXT d_kv=32 drops to 66.72 from 70.78).
• Qwen2.5-VL fine-tuning uses the LLaVA-NeXT data as a proxy because its original training data is not public, leaving a potential data gap.
• Evaluated only on 7-8B VLMs and eight benchmarks; LLaVA-1.5 excludes ChartQA and DocVQA.

Experimental setup

Datasets: LLaVA-1.5 visual instruction data LLaVA-NeXT instruction data

Benchmarks: AI2D GQA POPE SEED-Bench-IMG RealWorldQA MMBench ChartQA DocVQA

Metrics: accuracy average score across benchmarks KV cache memory reduction

Baselines: H2O TOVA Int4 Quanto Int4 HQQ Int2 HQQ S_2-norm partial-RoPE w/o modality decoupled w/o MD-SVD init w/o two-stage training

Builds on: MHA2MLA SVD-LLM V2 M-RoPE MLA

Closest papers by meaning

• A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMs (CVPR 2025 · similarity 0.91)
• A-VL: Adaptive Attention for Large Vision-Language Models (AAAI 2025 · similarity 0.91)
• Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference (AAAI 2025 · similarity 0.91)
• VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration (ICLR 2025 · similarity 0.91)
• TGV-KV: Text-Grounded KV Eviction for Vision-Language Models (ICML 2026 · similarity 0.90)
• Lifting the Veil on Visual Information Flow in MLLMs: Unlocking Pathways to Faster Inference (CVPR 2025 · similarity 0.90)