Research map / Large language models / Reasoning and chain of thought

When Reasoning Meets Compression: Understanding the Effects of LLMs Compression on Large Reasoning Models

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Benchmarks quantized, distilled, and pruned DeepSeek-R1 variants on four reasoning datasets and uses adapted difference-of-means and attribution patching to locate weights critical for reasoning, then uses the findings to protect final-layer MLP modules in 3-bit AWQ.

Problem

Compressed large reasoning models (LRMs) lose reasoning capability, but existing studies either do not compare quantization, distillation and pruning on hard reasoning tasks or lack interpretation of which weights matter for reasoning under compression.

Approach

Adapts difference-of-means steering vectors and attribution patching to score every linear module's causal importance for four reasoning behaviors, then tracks importance shifts between original and compressed models to reveal that the final-layer MLP up projection is critical and that current quantizers over-compress final-layer modules and MLP gate projections.

Key results

• Dynamically quantized 2.51-bit R1 reaches 84.8 average accuracy versus 83.1 for full R1 across AIME 2024, FOLIO and Temporal.
• Quantizing only the final-layer mlp.up_proj of R1-Distill-Llama-8B to 3-bit drops average accuracy by 16.3%.
• Protecting only final-layer MLP modules (about 2% of weights) in 3-bit AWQ raises average accuracy by 6.57%, outperforming 3-bit baselines by at least 4.77%.

Surprising findings

• Pruning hurts knowledge memorization on MuSiQue earlier than it hurts AIME 2024 reasoning, with R1-Distill-Llama-70B collapsing between 30% and 40% sparsity on MuSiQue.
• Interpreting the base Llama-3.1-8B instead of the distilled model would lead to a different conclusion about the most important module.

Limitations

• Interpretability analysis uses only 120 GPT-4o-annotated instances and a small MuSiQue subset of 100 questions.
• Pruning analysis restricted to 30% AlphaPruning on R1-Distill-Llama-8B because higher sparsity produces unusable outputs.
• Dynamic quantization of 2.51-bit R1 uses different calibration than other methods, so comparisons are not identical-calibration.

Experimental setup

Datasets: AIME 2024 FOLIO Temporal Sequences (BIG-Bench Hard) MuSiQue

Benchmarks: AIME 2024 FOLIO Temporal Sequences MuSiQue

Metrics: accuracy exact match F1

Baselines: AWQ GPTQ GPTAQ ANY4/3 SparseGPT AlphaPruning Unsloth dynamic quantization

Builds on: attribution patching difference of means AWQ GPTQ SparseGPT

Closest papers by meaning

• Think Silently, Think Fast: Dynamic Latent Compression of LLM Reasoning Chains (NeurIPS 2025 · similarity 0.91)
• Reasoning Path Compression: Compressing Generation Trajectories for Efficient LLM Reasoning (NeurIPS 2025 · similarity 0.91)
• Reasoning Models Can be Accurately Pruned Via Chain-of-Thought Reconstruction (ICLR 2026 · similarity 0.90)
• Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression (ICML 2025 · similarity 0.90)
• Towards Efficient Large Language Reasoning Models via Extreme-Ratio Chain-of-Thought Compression (ICML 2026 · similarity 0.90)
• R-KV: Redundancy-aware KV Cache Compression for Reasoning Models (NeurIPS 2025 · similarity 0.89)