Research map / Architectures and efficiency / Pruning and efficient tuning

MUSTAFAR: Promoting Unstructured Sparsity for KV Cache Pruning in LLM Inference

NeurIPS 2025 · Architectures and efficiency › Pruning and efficient tuning · approach: Layer-wise pruning of LLMs · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Mustafar, which prunes KV caches with per-token magnitude-based unstructured sparsity (up to 70%) and runs a bitmap-format sparse attention kernel that speeds up decode.

Problem

KV cache memory is a major decode bottleneck for long-context LLM inference, and prior KV cache pruning has been limited to structured patterns that lose accuracy at moderate sparsity.

Approach

Shows that removing structural constraints gives higher KV sparsity at preserved accuracy; per-token magnitude pruning works for both Key and Value caches, and a bitmap-based compressed format with a custom SpMV attention kernel turns unstructured sparsity into memory and throughput gains that offset pruning overhead.

Key results

• At 70% Key and Value sparsity, per-token magnitude pruning reaches 40.96 LongBench average on Llama-3-8B-Instruct versus 43.19 dense, and 40.95 on Mistral-7B-Instruct-v0.2 versus 42.65 dense.
• KV cache at 70% sparsity compresses to 45% of dense size, and 50% sparsity yields 65% compression.
• Llama-3 batch size 8 reaches 2.23x tokens/sec throughput over dense inference at batch size 6.

Measured effects reported in the paper

Surprising findings

• Per-token magnitude pruning of the Value cache matches or beats output-aware pruning, despite the Value cache's uniform magnitude distribution.
• Structured ThinK pruning collapses at 70% Key sparsity (LongBench average 26.55) while unstructured pruning keeps 41.55.

Limitations

• Batch size 1 shows lower throughput than dense inference due to GPU underutilization in the sparse kernel.
• Low-bit quantization is not supported by the kernel, so joint KIVI accuracy was measured on simulated sparse quantized caches.
• Sparsity is applied uniformly across all layers and heads; per-layer or per-head allocation is left for future work.

Experimental setup

Datasets: LongBench RULER

Benchmarks: LongBench RULER

Metrics: LongBench average score RULER accuracy KV cache compression ratio tokens/sec throughput normalized kernel latency

Baselines: ThinK KIVI H2O 2:4 semi-structured sparsity FlashAttention cuBLAS dense

Builds on: ThinK Coruscant FlashLLM FlashAttention KIVI

Closest papers by meaning

• RocketKV: Accelerating Long-Context LLM Inference via Two-Stage KV Cache Compression (ICML 2025 · similarity 0.91)
• SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference (NeurIPS 2025 · similarity 0.90)
• SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget (ICLR 2025 · similarity 0.90)
• Inference-Time Hyper-Scaling with KV Cache Compression (NeurIPS 2025 · similarity 0.90)
• KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference (ICML 2025 · similarity 0.89)
• ThinK: Thinner Key Cache by Query-Driven Pruning (ICLR 2025 · similarity 0.89)