Research map / Architectures and efficiency / Attention and transformers
Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Polar Sparsity shows that attention head sparsity stays stable under batching while MLP neuron sparsity collapses, and uses trained routers plus custom sparse GEMM and FlashAttention kernels to deliver up to 2.2x batched decoding throughput gains.
Problem
Contextual activation sparsity in LLMs does not scale to large batch sizes because the union of active MLP neurons across a batch approaches dense computation, limiting throughput gains in batched inference.
Approach
Shift sparsity focus from MLP neurons to attention heads: train lightweight routers to pick top-k attention heads (or GQA groups) per request, and run a Selective Head FlashAttention kernel that computes only active heads, while using dynamic per-layer top-k selective GEMM for MLP in OPT models.
Key results
- Polar Sparsity delivers up to 2.2x higher decoding throughput than dense on OPT-66B at batch size 64 (seq len 1920), and about 2x over DejaVu at large batch.
- LLaMA-2 models reach up to 1.85x throughput speedup at large batch; LLaMA-3.1-70B reaches 1.51x at 62.5% attention density.
- At critical thresholds, average zero-shot accuracy stays within about 1% of dense for most models (e.g., LLaMA 2 7B 0.608 dense vs 0.594 PS-0.5 in the reported table; LLaMA 3.1 70B 0.724 vs 0.712).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply Polar Sparsity at the reported critical threshold | OPT 6.7B · Zero-shot average (9 tasks) | accuracy: 0.531 -> 0.524 | Hurt |
| Apply Polar Sparsity at the reported critical threshold | OPT 66B · Zero-shot average (9 tasks) | accuracy: 0.570 -> 0.560 | Hurt |
| Apply Polar Sparsity at the reported critical threshold | LLaMA 2 7B · Zero-shot average (9 tasks) | accuracy: 0.608 -> 0.594 | Hurt |
| Apply Polar Sparsity at the reported critical threshold | LLaMA 2 13B · Zero-shot average (9 tasks) | accuracy: 0.647 -> 0.633 | Hurt |
| Apply Polar Sparsity at the reported critical threshold | LLaMA 3.1 70B · Zero-shot average (9 tasks) | accuracy: 0.724 -> 0.712 | Hurt |
| Apply Polar Sparsity at the reported critical threshold | Mistral 7B · Zero-shot average (9 tasks) | accuracy: 0.663 -> 0.657 | Hurt |
| Apply Polar Sparsity at the reported threshold | Mistral-7B-inst · MMLU PRO | accuracy: 0.247 -> 0.244 | Hurt |
| Apply Polar Sparsity at the reported threshold | Llama-3.1-8B-inst · MMLU PRO | accuracy: 0.409 -> 0.384 | Hurt |
| Apply Polar Sparsity at the reported threshold | Qwen-2.5-14B-inst · MMLU PRO | accuracy: 0.497 -> 0.457 | Hurt |
| Use Polar Sparsity instead of dense decoding | OPT models at large batch sizes · Batched decoding | throughput: 2.2× | Helped |
| Replace DejaVu-style activation sparsity with Polar Sparsity | OPT models at large batch sizes · Batched decoding | throughput: 2× | Helped |
| Apply attention head sparsity | LLaMA-2 models · Batched decoding | throughput: 1.85× | Helped |
Surprising findings
- Attention head sparsity is batch-invariant while MLP union neuron sparsity degrades quickly with batch size.
- Larger models show greater head sparsity; OPT-66B loses only about 5% perplexity at 30% head activation versus 37.4% for OPT-6.7B.
Limitations
- Benefits diminish at small batch sizes due to limited GPU workload.
- Top-k heads per layer are fixed rather than input-adaptive; GQA group sparsity is weaker and shows faster accuracy degradation.
- Evaluation limited to greedy decoding and contexts up to 16k tokens; beam search and speculative decoding not studied.
Experimental setup
Datasets: WikiText-2 COPA OpenBookQA PIQA RTE Winogrande HellaSwag MMLU ARC-easy ARC-challenge MMLU-PRO LongBench
Benchmarks: COPA OpenBookQA PIQA RTE Winogrande HellaSwag MMLU ARC-easy ARC-challenge MMLU-PRO LongBench
Metrics: zero-shot accuracy decoding throughput (tokens/s) decode latency perplexity speedup
Baselines: Dense baseline Deja Vu ReLUfication ProSparse CATS TEAL GRIFFIN R-Sparse ProbePruning
Builds on: FlashAttention Deja Vu TEAL ReLUfication
Closest papers by meaning
- R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference (ICLR 2025 · similarity 0.92)
- La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation (ICML 2025 · similarity 0.92)
- SparQ Attention: Bandwidth-Efficient LLM Inference (ICML 2024 · similarity 0.91)
- Learn To be Efficient: Build Structured Sparsity in Large Language Models (NeurIPS 2024 · similarity 0.91)
- MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (NeurIPS 2024 · similarity 0.91)
- Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.