Research map / Large language models / Reasoning and chain of thought

Prescriptive Scaling Reveals the Evolution of Language Model Capabilities

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Estimates high-quantile (0.98) sigmoid capability boundaries of post-trained open-weight LMs versus pre-training FLOPs, validates them temporally, and proposes a balanced I-optimal sampler that recovers near-full frontiers with ~20% of evaluation budget, releasing the Proteus-2k dataset.

Problem

Given a pre-training compute budget, what downstream accuracy is attainable after contemporary post-training, and how stable is that mapping as training recipes evolve; evaluating every checkpoint across benchmarks is expensive.

Approach

Model attainable performance as the conditional 0.98-quantile of post-trained benchmark scores vs log pre-training FLOPs using a smoothed pinball loss with a monotone saturating sigmoid, validate by fitting on earlier periods and testing on later ones, and select evaluations via a bin-balanced I-optimal design.

Key results

• Sigmoid boundary achieves OOD calibration error 2.21% vs 3.60% for the constant baseline, matching I-spline pinball loss in ID (4.08e-3 vs 4.00e-3).
• At 1e24 FLOPs the estimated attainable accuracies are 0.828 on IFEval and 0.539 on MATH Lvl 5.
• Balanced I-optimal design recovers near-full-data boundaries using about 20% of the parameter-weighted evaluation budget, and about 5% on GPQA and MUSR.

Surprising findings

• Pretrained checkpoints are frequently locally non-monotone in compute, while post-trained boundaries are consistently monotone.
• No statistically significant evidence (p=0.15) of AIME-2025 score inflation relative to MATH-500 after release date.

Limitations

• Observational data from public leaderboards; boundaries are conditional on the observed model population and may be underestimated if unrepresented recipes exist.
• MATH Lvl 5 shows under-coverage on the earliest split, indicating non-stationarity.
• Compute values for many models rely on estimated base-model pretraining FLOPs.

Experimental setup

Datasets: Open LLM Leaderboard v1 Open LLM Leaderboard v2 Epoch AI frontier model data LifeArchitect.AI

Benchmarks: IFEval BBH MATH Lvl 5 GPQA MUSR MMLU-PRO AIME 2025 MATH-500

Metrics: smoothed pinball loss coverage error calibration error R^2 of logit regression

Baselines: constant boundary binwise constant I-spline

Builds on: Open LLM Leaderboard pinball loss quantile regression I-spline I-optimal design

Closest papers by meaning

• Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.90)
• A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.90)
• Algorithmic progress in language models (NeurIPS 2024 · similarity 0.90)
• Observational Scaling Laws and the Predictability of Langauge Model Performance (NeurIPS 2024 · similarity 0.90)
• The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs (ICLR 2026 · similarity 0.89)
• Predicting Emergent Abilities with Infinite Resolution Evaluation (ICLR 2024 · similarity 0.89)