Research map / Architectures and efficiency / Quantization

CASP: Compression of Large Multimodal Models Based on Attention Sparsity

CVPR 2025 · Architectures and efficiency › Quantization · approach: KV cache quantization · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CASP, a fine-tuning-free post-training compression method for large multimodal models that combines data-aware low-rank decomposition of Wq/Wk (motivated by attention sparsity) with layer-wise optimal bit allocation, boosting 2-bit AQLM and QuIP# on LMM and LLM benchmarks.

Problem

Low-bit (2-bit) post-training compression of large multimodal models is under-explored, and 2-bit quantization of LLMs/LMMs causes large accuracy drops without costly fine-tuning.

Approach

The sparsity of attention maps in LMMs (dominated by text tokens since visual tokens get little attention) bounds the compression error of Wq and Wk, so these matrices can be aggressively low-rank compressed; the freed budget is redistributed via an entropy-regularized optimal per-layer bit allocation before applying any vector quantizer.

Key results

• On LLaVA1.5-7B, CASP improves 2-bit AQLM by 38.7% and 2-bit QuIP# by 5.7% in average relative improvement over the image-language benchmarks.
• On LLaVA-Next-Video-7B, CASP improves 2-bit AQLM by 159.0% and QuIP# by 21.9% on video-language benchmarks.
• On Llama2-7B (language-only), CASP yields 11.2% improvement over AQLM and 2.7% over QuIP# in average relative improvement on C4/WikiText2 perplexity.

Measured effects reported in the paper

Surprising findings

• Using text-only calibration data (C4, RedPajama) gave lower perplexity than multimodal LLaVA-Instruct-150K calibration for CASP-AQLM on LLaVA1.5-7B.
• Compression of Wq and Wk tends to be less harmful for models with more visual tokens, so the LLaVA-Next-Video model with ~4600 visual tokens showed the smallest drop, while Llama2 with no visual tokens showed the largest.

Limitations

• Bit allocation is per layer rather than per weight matrix, and non-integer optimal bits must be rounded to integers.
• Baselines are evaluated without fine-tuning for fair comparison, so absolute results differ from published AQLM/QuIP# numbers.
• Relative gains are small for QuIP# because the baseline already nearly matches the original model.

Experimental setup

Datasets: RedPajama C4 LLaVA-Instruct-150K WikiText2

Benchmarks: LiveBench LLaVA-Bench-Wilder LLaVA-Bench-COCO SEED-Bench MMMU ScienceQA MME MMBench VideoDetailCaption VideoChatGPT NextQA VideoMME

Metrics: perplexity accuracy exact match ROUGE-L GPT score

Baselines: GPTQ AQLM QuIP#

Builds on: AQLM QuIP# GPTQ SVD-LLM Block Influence

Closest papers by meaning

• Compressing Large Language Models using Low Rank and Low Precision Decomposition (NeurIPS 2024 · similarity 0.88)
• Compression via Pre-trained Transformers: A Study on Byte-Level Multimodal Data (ICML 2025 · similarity 0.88)
• Compressing LLMs: The Truth is Rarely Pure and Never Simple (ICLR 2024 · similarity 0.87)
• BitStack: Any-Size Compression of Large Language Models in Variable Memory Environments (ICLR 2025 · similarity 0.87)
• Breaking the Compression Ceiling: Data-Free Pipeline for Ultra-Efficient Delta Compression (NeurIPS 2025 · similarity 0.87)
• Compressing Large Language Models by Joint Sparsification and Quantization (ICML 2024 · similarity 0.87)