Research map / Large language models / Reasoning and chain of thought

SIRIUS : Contexual Sparisty with Correction for Efficient LLMs

NeurIPS 2024 · Large language models › Reasoning and chain of thought · approach: Efficient LLM serving · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Sirius corrects contextually sparse LLMs on reasoning tasks by letting the full model periodically verify and rewrite the sparse model's tokens, recovering accuracy while keeping most of the speedup.

Problem

Contextual sparsity (CS) methods for LLM inference are training-free and efficient but severely degrade performance on reasoning, math, coding, and knowledge-based generation tasks.

Approach

Sparse and full models share weights and KV cache; the full model periodically verifies a chunk of sparse-generated tokens in parallel, rewrites the shared KV cache, rolls back rejected tokens and interleaves its own token, with a tree of top-k alternatives to lengthen the accepted advance length.

Key results

• On GSM8K with Llama-3-8B-Instruct FSparse, Sirius raises accuracy from 58.68% to 72.78% at effective density 0.807 (AAL 15.37/16).
• On HumanEval with Llama-3-8B-Instruct FSparse, Sirius raises pass@1 from 45.7% to 61.6% (AAL 15.42/16).
• Wall-clock latency on A100 for Llama-3-8B-Instruct is about 0.83-0.88x of full model, and offloaded Llama-3-70B-Instruct latency is 0.64x of full with accuracy 0.87 vs 0.90 full.

Measured effects reported in the paper

Surprising findings

• Sparse models' token entropy at error positions is not elevated, so self-confidence signals cannot reliably trigger correction.
• Well-trained models (Llama-3) degrade more under contextual sparsity than Llama-2 models of similar size.

Limitations

• Relies on rollback for correction, which the authors note is not the most efficient choice since errors are uniformly distributed across chunk positions.
• Correction period is a fixed hyperparameter because sparse-model confidence cannot reliably trigger correction.
• Large-model (70B) tree experiments were not run due to computational limits.

Experimental setup

Datasets: C4 GSM8K AQuA-RAT-COT CSQA StrategyQA Date Sports HumanEval MBPP+

Benchmarks: GSM8K AQuA-RAT-COT CSQA StrategyQA Date Sports HumanEval MBPP+ MMLU-FLAN-COT CNN/DailyMail CoQA TruthfulQA

Metrics: accuracy pass@1 average advance length effective density wall-clock latency ratio

Baselines: Griffin (coarse-grained contextual sparsity) CATS (fine-grained contextual sparsity) Speculative Decoding

Builds on: Griffin CATS Speculative Decoding Deja Vu

Closest papers by meaning

• R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference (ICLR 2025 · similarity 0.87)
• Learn To be Efficient: Build Structured Sparsity in Large Language Models (NeurIPS 2024 · similarity 0.86)
• La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation (ICML 2025 · similarity 0.86)
• Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsity (NeurIPS 2025 · similarity 0.86)
• Training-Free Activation Sparsity in Large Language Models (ICLR 2025 · similarity 0.86)
• Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference (NeurIPS 2025 · similarity 0.86)