Research map / Large language models / Reasoning and chain of thought
Not All Bits Are Equal: How Model Scale Changes Memory-Optimal Reasoning
Junhyuck Kim, Ethan Ewer, Taehong Moon, Jongho Park, Dimitris Papailiopoulos
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that the universal 4-bit quantization prescription fails for reasoning models, where KV cache can dominate memory, and that the memory-optimal bit-width depends on model scale.
Problem
Identifying the memory-optimal quantization bit-width for reasoning models, where long chain-of-thought generation makes KV cache rather than weights the dominant memory cost.
Approach
Challenges the prior finding that 4-bit quantization is memory-optimal across scales by showing that for reasoning models the optimal choice is scale-dependent, since KV cache can dominate memory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| spend memory on larger effective model size instead of longer generation, for models below 8-bit 4B (ablation) | Qwen3 reasoning models · LiveCodeBench | accuracy (%): 42.5 -> 61.7 (1.7B 16-bit 22k vs 4B 8-bit 14k, ~6.2 GB) | Improved |
| spend memory on longer generation instead of weights, for models at or above 8-bit 4B (ablation) | Qwen3-14B 4-bit vs Qwen3-4B 16-bit · LiveCodeBench | accuracy (%): 26.8 -> 65.00 (4B 16-bit 22k vs 14B 4-bit 6k, ~10.5 GB) | Improved |
| use 8-bit or 16-bit weights instead of 4-bit for math and code (ablation) | Qwen3 reasoning models · AIME25 | accuracy | Improved |
| use 4-bit weights for knowledge-intensive reasoning (ablation) | Qwen3 reasoning models · GPQA-Diamond | accuracy per memory | Improved |
| add parallel scaling (majority voting) to models below 8-bit 4B (ablation) | Qwen3 reasoning models · AIME25 | accuracy per memory | Hurt |
| add parallel scaling (majority voting) to models at or above 8-bit 4B (ablation) | Qwen3 reasoning models · AIME25 | accuracy per memory | Improved |
| add external PRM verifier (ActPRM-X, 13.28 GB) with Best-of-N (ablation) | Qwen3 reasoning models · AIME25 | accuracy per memory | No clear effect |
| add KV cache eviction (R-KV) or quantization (ablation) | Qwen3 reasoning models · AIME25 | accuracy per memory | Improved |
| use KV eviction instead of KV quantization for small effective sizes (ablation) | Qwen3 reasoning models · AIME25 | accuracy per memory | Improved |
| use 2-bit KV cache quantization (ablation) | Qwen3 reasoning models · AIME25 | accuracy | Hurt |
| use AWQ or FP8 weight quantization instead of GPTQ (ablation) | Qwen3 reasoning models · AIME25 | accuracy per memory | No clear effect |
| use 4-bit weights for latency or throughput (ablation) | Qwen3 reasoning models · AIME25 | latency/throughput vs accuracy | Hurt |
Surprising findings
- The universal prescription of 4-bit quantization as memory-optimal fails for reasoning models, where KV cache rather than model size can dominate memory.
Closest papers by meaning
- Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning (NeurIPS 2025 · similarity 0.89)
- SCALE: Selective Resource Allocation for Overcoming Performance Bottlenecks in Mathematical Test-time Scaling (AAAI 2026 · similarity 0.89)
- ArborKV: Structure-Aware KV Cache Management for Scaling Tree-based LLM Reasoning (ICML 2026 · similarity 0.89)
- Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving (ICLR 2025 · similarity 0.89)
- TrimR: Verifier-based Training-Free Thinking Trimming for Efficient Test-Time Scaling (ICLR 2026 · similarity 0.88)
- When Reasoning Meets Compression: Understanding the Effects of LLMs Compression on Large Reasoning Models (ICLR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
