AI Paper Atlas

Research map / Large language models / Reasoning and chain of thought

Not All Bits Are Equal: How Model Scale Changes Memory-Optimal Reasoning

Junhyuck Kim, Ethan Ewer, Taehong Moon, Jongho Park, Dimitris Papailiopoulos

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: KV cache compression

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that the universal 4-bit quantization prescription fails for reasoning models, where KV cache can dominate memory, and that the memory-optimal bit-width depends on model scale.

Problem

Identifying the memory-optimal quantization bit-width for reasoning models, where long chain-of-thought generation makes KV cache rather than weights the dominant memory cost.

Approach

Challenges the prior finding that 4-bit quantization is memory-optimal across scales by showing that for reasoning models the optimal choice is scale-dependent, since KV cache can dominate memory.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
spend memory on larger effective model size instead of longer generation, for models below 8-bit 4B (ablation)Qwen3 reasoning models · LiveCodeBenchaccuracy (%): 42.5 -> 61.7 (1.7B 16-bit 22k vs 4B 8-bit 14k, ~6.2 GB)Improved
spend memory on longer generation instead of weights, for models at or above 8-bit 4B (ablation)Qwen3-14B 4-bit vs Qwen3-4B 16-bit · LiveCodeBenchaccuracy (%): 26.8 -> 65.00 (4B 16-bit 22k vs 14B 4-bit 6k, ~10.5 GB)Improved
use 8-bit or 16-bit weights instead of 4-bit for math and code (ablation)Qwen3 reasoning models · AIME25accuracyImproved
use 4-bit weights for knowledge-intensive reasoning (ablation)Qwen3 reasoning models · GPQA-Diamondaccuracy per memoryImproved
add parallel scaling (majority voting) to models below 8-bit 4B (ablation)Qwen3 reasoning models · AIME25accuracy per memoryHurt
add parallel scaling (majority voting) to models at or above 8-bit 4B (ablation)Qwen3 reasoning models · AIME25accuracy per memoryImproved
add external PRM verifier (ActPRM-X, 13.28 GB) with Best-of-N (ablation)Qwen3 reasoning models · AIME25accuracy per memoryNo clear effect
add KV cache eviction (R-KV) or quantization (ablation)Qwen3 reasoning models · AIME25accuracy per memoryImproved
use KV eviction instead of KV quantization for small effective sizes (ablation)Qwen3 reasoning models · AIME25accuracy per memoryImproved
use 2-bit KV cache quantization (ablation)Qwen3 reasoning models · AIME25accuracyHurt
use AWQ or FP8 weight quantization instead of GPTQ (ablation)Qwen3 reasoning models · AIME25accuracy per memoryNo clear effect
use 4-bit weights for latency or throughput (ablation)Qwen3 reasoning models · AIME25latency/throughput vs accuracyHurt

Surprising findings

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy