Research map / Architectures and efficiency / Quantization
QSVD: Efficient Low-rank Approximation for Unified Query-Key-Value Weight Compression in Low-Precision Vision-Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Applies SVD jointly to concatenated query/key/value weights of VLMs, allocates ranks globally via a Fisher-like importance score, and adds outlier-suppressing rotations plus learned beta and W8A8/W4A4 quantization to shrink KV cache and compute.
Problem
VLM inference is costly in compute and KV-cache memory, which limits deployment on resource-constrained devices; prior SVD compression works per matrix and rarely combines with quantization for VLMs.
Approach
Concatenate W_Q, W_K, W_V into one matrix and factor it with a shared down-projection so only a rank-r latent C_qkv is cached; rank is allocated globally across layers using a per-singular-value importance score derived from gradients, and a learned exponent beta plus orthogonal rotations suppress outliers before low-bit quantization.
Key results
- Under W8A8 on LLaVA-v1.5 13B, QSVD reaches 72.12% on ScienceQA vs 71.78% FP16 while using 50% QKV weight/compute and 18.75% KV cache.
- At W4A4 on SmolVLM 2B, QSVD-noQ-style low-rank models keep over 70% ScienceQA accuracy where ASVD and SVD-LLM collapse (ASVD 7.88% at 90% rank).
- On RTX 4070 (12GB), QSVD W8A8 at 50-70% rank retention reports up to 13.1x speedup over the FP16 baseline in layer-wise latency evaluation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use joint QKV SVD with W8A8 quantization rather than QuaRot | LLaVA-v1.5 7B · VizWiz | accuracy: 51.93% -> 54.06% | Helped |
| Use joint QKV SVD with W8A8 quantization rather than Duquant | LLaVA-v1.5 13B · ScienceQA | accuracy: 69.66% -> 72.12% | Helped |
| Use joint QKV SVD with W8A8 quantization rather than QSVDLLM | LLaVA-v1.5 13B · VizWiz | accuracy: 56.32% -> 55.42% | Hurt |
| Use joint QKV SVD without quantization rather than SVDLLM | LLaVA-v1.5 13B · VizWiz | accuracy: 49.37% -> 55.79% | Helped |
| Compress with QSVD-noQ at R1=70% | LLaVA-v1.5 13B · HallusionBench | Overall: 26.70% -> 30.30% | Helped |
| Apply QSVD-noQ compression | LLaVA-v1.5 7B on RTX 4070 · 4K token length, batch size 1 | speedup: 2.1× | Helped |
| Add W8A8 quantization to QSVD | LLaVA-v1.5 7B on RTX 4070 · 4K token length, batch size 1 | speedup: 13.1× | Helped |
| Use uniform rank allocation across VLM blocks instead of cross-layer rank allocation (ablation) | QSVD-noQ on LLaVA-v1.5 13B · ScienceQA-IMG, R1 53.3% / R2 20.0% | accuracy: 71.74% -> 70.40% | Hurt |
| Use Fisher-information rank allocation instead of cross-layer rank allocation (ablation) | QSVD-noQ on LLaVA-v1.5 13B · ScienceQA-IMG, R1 53.3% / R2 20.0% | accuracy: 71.74% -> 70.60% | Hurt |
| Learn beta rather than fixing it at 0.0 (ablation) | QSVD on LLaVA-v1.5 7B · Science QA W4A4 | accuracy: 54.53% -> 55.16% | Helped |
| Learn beta rather than fixing it at 0.4 (ablation) | QSVD on LLaVA-v1.5 7B · Science QA W4A4 | accuracy: 54.83% -> 55.16% | Helped |
| Learn beta rather than fixing it at 0.8 (ablation) | QSVD on LLaVA-v1.5 7B · Science QA W4A4 | accuracy: 6.09% -> 55.16% | Helped |
Surprising findings
- Low-rank truncation of LLaVA-v1.5 13B slightly exceeds FP16 accuracy on VizWiz (55.79% vs 53.63%), which the authors attribute to hallucination reduction but leave unverified.
- Fixed beta values of 0.4 or 0.8 degrade W4A4 accuracy sharply (e.g., 6.09% at beta 0.8 on LLaVA-v1.5 7B), showing sensitivity to the power applied to singular values.
Limitations
- Only per-layer beta is learned and only self-attention is compressed; joint optimization across all blocks is left to future work.
- Calibration uses only 256 ScienceQA samples and evaluation covers a limited set of benchmarks.
- The hallucination-reduction explanation for surpassing FP16 accuracy is not verified.
Experimental setup
Datasets: ScienceQA training set (256 samples for calibration)
Benchmarks: ScienceQA-IMG VizWiz SEED-Bench-IMG HRBench-4K HallusionBench
Metrics: accuracy hardware cost ratio R1 (weights/FLOPs) KV cache ratio R2 speedup latency
Baselines: ASVD SVD-LLM QuaRot DuQuant QVLM QASVD QSVDLLM
Builds on: ASVD SVD-LLM QuaRot DuQuant Multi-Head Latent Attention
Closest papers by meaning
- WSVD: Weighted Low-Rank Approximation for Fast and Efficient Execution of Low-Precision Vision-Language Models (ICLR 2026 · similarity 0.95)
- VL-Cache: Sparsity and Modality-Aware KV Cache Compression for Vision-Language Model Inference Acceleration (ICLR 2025 · similarity 0.90)
- VLM-PTQ: Efficient Post-Training Quantization for Large Vision-Language Models (CVPR 2026 · similarity 0.89)
- SPHQuant: Efficient extreme low bit weight quantization for Vision-Language Models (NeurIPS 2026 · similarity 0.89)
- Rethinking Token Reduction for Large Vision-Language Models (CVPR 2026 · similarity 0.89)
- Benchmarking and Enhancing VLM for Compressed Image Understanding (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.