Research map / Large language models / Language, safety and interpretability
Predicting Emergent Abilities with Infinite Resolution Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PassUntil, a sampling-until-pass evaluation with theoretically infinite resolution, to fit a task scaling law that predicts a 2.4B model's code generation performance and to characterize "accelerated emergence" with a multiple-circuits hypothesis.
Problem
Task performance of LLMs is not predictable from scaling: coarse evaluation resolution makes small-model gains look like zero, so emergent abilities appear discontinuous and task scaling cannot be forecast.
Approach
Estimate per-instance pass probability by repeated random sampling until r samples pass (up to 1e4-1e5 draws), yielding PassUntil scores that reveal tiny performance; then fit log(-log PU) linearly in log N (task scaling law) at dataset or instance level, and classify emergence curves by convexity.
Key results
- Predicted the 2.4B model's HumanEval PassUntil with 0.05% deviation from the actual value before training it.
- Date Understanding 2.4B prediction deviates by 1.7%, showing predictive accuracy across tasks.
- 5 of 8 Unnatural In-context Learning tasks show concave (accelerated emergence) log(-log PU) vs log N curves, which the authors attribute to a multiple-circuits hypothesis.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PassUntil with extensive random decoding samples to evaluate task performance | Small language models · HumanEval, Emoji Movie, and Date Understanding | task performance predictability: 0.05% deviation; 1.7% difference | Helped |
| Fit PassUntil separately for each instance and aggregate the estimates (ablation) | Dataset-level PassUntil scaling fit · HumanEval, Series 2 | prediction precision | Helped |
| Use test loss on ground-truth answers to assist hard-instance PassUntil predictions (ablation) | Instance-level PassUntil prediction · HumanEval and Date Understanding | prediction accuracy: 0.05% difference; 1.7% difference | Helped |
| Scale model size using the task scaling law fit | Series 1 and Series 2 language models · HumanEval, Emoji Movie, and Date Understanding | PassUntil scaling: 0.05% deviation | Helped |
| Fit a two-circuit weighted combination instead of a single standard scaling curve (ablation) | Unnatural In-context Learning tasks · BigBench UICL | fit to observed performance scaling curve | Helped |
| Model task performance as multi-step reasoning | Tasks whose individual reasoning steps follow scaling-law growth · Unnatural In-context Learning | scaling-curve growth | Hurt |
| Use multiple neural circuits as alternative ways to solve a task | Unnatural In-context Learning tasks · BigBench UICL | fit to observed performance scaling curve | Helped |
| Increase model size across the tested UICL tasks | 0.03B to 2.4B model series · BigBench UICL | growth speed: 5/8; 3/8 | Mixed |
Surprising findings
- Small models show non-zero PassUntil performance on the order of 1e-5 that conventional evaluation reports as zero.
- Emoji Movie's apparent emergence is partly driven by distracting instances whose common-word answers let small models guess correctly.
Limitations
- Only applicable to tasks where random baseline is zero; multiple-choice tasks have non-zero random baselines.
- Evaluated only up to 2.4B parameters; no extrapolation to 10B+ models.
- Multiple-circuits hypothesis is only loosely tested by curve fitting, not mechanistically validated.
Experimental setup
Datasets: StarCoder The Pile HumanEval BIG-bench Emoji Movie BIG-bench Date Understanding BIG-bench Unnatural In-context Learning
Benchmarks: HumanEval Emoji Movie Date Understanding Unnatural In-context Learning
Metrics: PassUntil score pass rate prediction deviation
Baselines: beam search greedy/fixed-K random sampling evaluation
Builds on: Chinchilla scaling law Emergent abilities Grokking circuits
Closest papers by meaning
- U-shaped and Inverted-U Scaling behind Emergent Abilities of Large Language Models (ICLR 2025 · similarity 0.91)
- Observational Scaling Laws and the Predictability of Langauge Model Performance (NeurIPS 2024 · similarity 0.90)
- Understanding Emergent Abilities of Language Models from the Loss Perspective (NeurIPS 2024 · similarity 0.90)
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.90)
- How Do Large Language Monkeys Get Their Power (Laws)? (ICML 2025 · similarity 0.90)
- Training on the Test Task Confounds Evaluation and Emergence (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.