Research map / Architectures and efficiency / RNNs and normalization
Kinetics: Rethinking Test-Time Scaling Law
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Kinetics scaling law for test-time scaling that adds memory (KV cache) access to the cost model, and Sparse Kinetics showing block top-k sparse attention greatly improves cost-accuracy tradeoffs of Best-of-N and Long-CoT on Qwen3 and DeepSeek-R1-Distill models.
Problem
Existing test-time scaling laws use FLOPs and compute-optimality, ignoring memory access (KV cache) costs that dominate wall-clock latency in long-CoT and Best-of-N inference, leading to suboptimal choices of model size and test-time compute.
Approach
Derives an eFLOPs cost model combining compute and memory access weighted by arithmetic intensity, showing attention's quadratic KV cost dominates long generations so small models are overestimated; then shows sparse (block top-k) attention reduces the quadratic term to linear, reshaping the scaling law to favor sparsity plus test-time compute.
Key results
- Sparse attention with oracle top-k improves problem-solving rates by up to 60 points in the low-cost regime and over 5 points in the high-cost regime on AIME24 and LiveCodeBench.
- Block top-k attention yields up to 11.2–26.2x speedup on H200 GPUs, with Qwen3-0.6B reaching ~25.9–26.2x throughput increase.
- Under the Kinetics law, the critical model size is about 14B for Qwen3 (7B for DeepSeek-R1-Distill-Qwen); dense models need over 10x more eFLOPs to match sparse solving rates.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use sparse attention instead of dense attention | Qwen3 models with test-time scaling · AIME24 | problem-solving rate: up to 60 points in the low-cost regime and over 5 points in the high-cost regime | Helped |
| Use block top-k attention instead of dense attention | Qwen3 models with test-time scaling · AIME24 | resources to reach equivalent accuracy: 8.58× fewer resources | Helped |
| Use block top-k attention instead of dense attention | Qwen3-0.6B · Inference throughput | throughput: 25.9\sim26.2\times | Helped |
| Replace top-k KV selection with local attention (ablation) | Sparse attention models · AIME24 | problem-solving rate | Hurt |
| Evaluate with max cost model instead of additive cost model (ablation) | Sparse attention versus dense attention · AIME24, Long-CoTs | accuracy gap: 47.5 points ... to 52.8 points | Helped |
| Evaluate with max cost model instead of additive cost model (ablation) | Sparse attention versus dense attention · AIME24, Best-of-N | accuracy gap: 65 points ... to 69.4 points | Helped |
| Increase model size rather than extend generation length | Qwen3 models below 14B · AIME24, Long-CoTs | Pass@1 | Helped |
| Extend CoTs beyond 10K tokens | Qwen3-14B and Qwen3-32B · AIME24, Long-CoTs | Pass@1: 10K tokens | Helped |
| Increase model size rather than add reasoning trials | Qwen3 models below 14B · Best-of-N reasoning | problem-solving rate | Helped |
| Use sparse attention instead of dense attention | Qwen3-30B-A3B · AIME24, low-cost regime | problem-solving rate: 42--53 percentage points | Helped |
| Allocate additional compute to generated tokens rather than KV budget | Qwen3-8B with sparse attention · Best-of-N | optimal generated tokens per cost doubling: KV budget increases by a factor of 1.18×, while the total number of generated tokens increases by 1.74× | Helped |
Surprising findings
- Smaller models (0.6B–4B) are substantially overestimated by FLOPs-based scaling laws; the 0.6B model is only on the Pareto frontier at negligible accuracy.
- Doubling model parameters only increases KV cache by about 1.18x for Qwen3, so scaling model size is favored over longer generation at low budgets.
Limitations
- Evaluation focused on Qwen3 and DeepSeek-R1-Distilled-Qwen; training and post-training effects not explored.
- Cost analysis assumes cloud serving with large batch sizes; local deployment with limited VRAM may differ.
- Sparse results use oracle top-k as upper bound; block top-k speedups are estimated from per-layer latency rather than full end-to-end serving.
Experimental setup
Datasets: AIME24 AIME25 LiveCodeBench
Benchmarks: AIME24 AIME25 LiveCodeBench
Metrics: Pass@1 Pass@K solving rate eFLOPs cost accuracy vs cost Pareto frontier throughput speedup
Baselines: FLOPs-based test-time scaling law dense attention local attention oracle top-k attention
Builds on: Chinchilla scaling laws Best-of-N sampling FlashInfer vLLM
Closest papers by meaning
- ATTS: Asynchronous Test-Time Scaling via Conformal Prediction (ICLR 2026 · similarity 0.90)
- Test-Time Scaling in the Wild: Why Exploitation, Not Exploration, Is the Bottleneck (NeurIPS 2026 · similarity 0.89)
- SCALE: Selective Resource Allocation for Overcoming Performance Bottlenecks in Mathematical Test-time Scaling (AAAI 2026 · similarity 0.89)
- Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verification (ICML 2025 · similarity 0.88)
- Optimal Aggregation of LLM and PRM Signals for Efficient Test-Time Scaling (ICLR 2026 · similarity 0.88)
- Power Lines: Scaling laws for weight decay and batch size in LLM pre-training (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.