Research map / Large language models / Reasoning and chain of thought
Observational Scaling Laws and the Predictability of Langauge Model Performance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes observational scaling laws that build capability-vs-compute relationships from ~100 public models' standard benchmark scores via PCA, predicting emergent abilities, agent performance, and post-training gains (CoT, self-consistency).
Problem
Scaling analyses for downstream capabilities, agentic behaviour, and post-training interventions are costly because they require training many models across scales, and open model families are typically trained at only 3-5 scales.
Approach
Model families differ only in compute efficiency, so a low-dimensional capability space extracted by PCA from standard benchmarks (top 3 PCs explain ~97% of variance) relates log-linearly to compute within each family and linearly to downstream metrics across families; a sigmoidal regression then predicts target metrics from weaker models without training new ones.
Key results
- Top 3 principal components explain ~97% of variance across benchmarks, with PC-1 alone explaining nearly 80%.
- Observational scaling law predicts GPT-4 and Claude-2 agent performance on AgentBench/AgentBoard from weaker models with >10% gap.
- Predicts CoT and self-consistency gains on GSM8K from models up to Llama-2 7B scale.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PC capability measures instead of training FLOPs to predict emergent-task performance | Observational scaling laws · Four BigBench emergent tasks | test MSE | Helped |
| Use PC capability measures to extrapolate from weaker models to held-out stronger models | Observational scaling laws · AgentBench | Overall Score prediction | Helped |
| Use PC capability measures to extrapolate from weaker models to held-out stronger models | Observational scaling laws · AgentBoard | Average Success Rate prediction | Helped |
| Use V-optimality to select a model subset rather than selecting models randomly | Observational scaling analyses · Post-training prediction tasks (Naive, CoT, CoT + SC) | geometric average of test MSEs | Helped |
| Use 12 V-optimality-selected models instead of the full set of 47 models | Observational scaling analyses · Post-training prediction tasks | prediction accuracy | No clear effect |
| Apply Chain-of-Thought prompting rather than naive few-shot prompting | Pretrained base models · GSM8K | accuracy scaling trend | Helped |
| Apply Self-Consistency to Chain-of-Thought prompting | Pretrained base models · GSM8K | accuracy scaling trend | Mixed |
| Use three principal components to represent benchmark capabilities (ablation) | PCA capability measures · Collected base-model benchmarks | explained variance | Helped |
Surprising findings
- Phi family is a clear outlier in compute efficiency, benefiting from CoT earlier but scaling less rapidly.
- Programming capability (HumanEval) is a key driver of agent performance.
Limitations
- Primarily applicable to post-training scaling, not pretraining hyperparameter design.
- Focuses on few-shot prompting and basic prompting interventions rather than fine-tuning or heavier inference compute.
- Does not account for benchmark contamination or heterogeneity within model families.
Experimental setup
Datasets: MMLU ARC-C HellaSwag Winogrande GSM8K TruthfulQA XWinograd HumanEval BigBench AgentBench AgentBoard
Benchmarks: BigBench emergent tasks AgentBench AgentBoard GSM8K CoT and self-consistency BigBench-Hard
Metrics: mean squared error
Baselines: training FLOPs scaling model size scaling
Builds on: compute scaling laws sigmoidal scaling laws Chain-of-Thought Self-Consistency
Closest papers by meaning
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.92)
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.92)
- Scaling Inference-Efficient Language Models (ICML 2025 · similarity 0.92)
- Resolving Discrepancies in Compute-Optimal Scaling of Language Models (NeurIPS 2024 · similarity 0.91)
- Scaling Laws for Precision (ICLR 2025 · similarity 0.90)
- Predicting Emergent Abilities with Infinite Resolution Evaluation (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.