Research map / Vision-language and multimodal / Multimodal LLMs

WSVD: Weighted Low-Rank Approximation for Fast and Efficient Execution of Low-Precision Vision-Language Models

ICLR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Attention-based visual token pruning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes WSVD, a per-head SVD scheme for VLM key/value projections with Fisher-weighted local fine-tuning, local quantization-aware training, and a fused Triton kernel integrated into flash decoding, reporting over 1.8x decoding speedup at comparable accuracy.

Problem

SVD-based low-rank compression of vision-language models often fails to reduce decoding latency because reconstructing K and V from shared low-rank latents forces repeated memory access, and truncation treats all weight elements equally, hurting accuracy.

Approach

Apply SVD per attention head so each head reconstructs K/V from its own latent, weight the SVD truncation error by element-wise Fisher information with local fine-tuning and QAT, and fuse low-rank reconstruction into the flash decoding kernel.

Key results

• WSVD-noQ achieves over 1.8x decoding speedup over Flash Decoding on LLaVA-Next 7B at 8K sequence length on RTX 4090 (2.92 ms vs 1.66 ms).
• Under W8A8 quantization with rho1=50%, WSVD averages 67.10% across LLaVA variants and ScienceQA/SEED, versus 66.07% for QSVD and about 68.23% for FP16.
• On LLaVA-Next 13B, WSVD-noQ at rho1=70% reaches 73.57% on ScienceQA-IMG, exceeding the FP16 baseline of 73.23%.

Surprising findings

• Naive SVD of K and V can make decoding latency higher than uncompressed flash decoding because reconstruction re-reads the shared latent.
• Low-rank WSVD-noQ slightly outperforms FP16 on LLaVA-Next 13B ScienceQA-IMG at moderate compression, which the authors suggest may reflect mitigated hallucination.

Limitations

• Accuracy evaluation is limited to ScienceQA-IMG, SEED-Bench, OCRBench, and HRBench-4K, with the 13B and extra-model results mostly in appendix.
• The paper notes that reproducing 13B experiments requires high-end GPUs.
• Per-head SVD amplifies approximation errors, making accuracy harder to control than full-matrix SVD.

Experimental setup

Datasets: ScienceQA

Benchmarks: ScienceQA-IMG SEED-Bench OCRBench HRBench-4K

Metrics: accuracy decoding latency (ms) speedup

Baselines: ASVD SVD-LLM QSVD DuQuant QVLM QASVD Palu Flash Decoding DobiSVD

Builds on: SVD-LLM QSVD Palu FlashAttention Flash Decoding QuaRot

Closest papers by meaning

• QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Models (NeurIPS 2025 · similarity 0.95)
• SPHQuant: Efficient extreme low bit weight quantization for Vision-Language Models (NeurIPS 2026 · similarity 0.88)
• ZIP: An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models (ICLR 2025 · similarity 0.87)
• VLM-PTQ: Efficient Post-Training Quantization for Large Vision-Language Models (CVPR 2026 · similarity 0.87)
• Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives (ICLR 2025 · similarity 0.86)
• Connecting the Dots: Collaborative Fine-tuning for Black-Box Vision-Language Models (ICML 2024 · similarity 0.86)