Research map / Large language models / Language, safety and interpretability
Understanding Emergent Abilities of Language Models from the Loss Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Pre-trains 30+ Transformer LMs (300M to 32B) and analyzes LLaMA and Pythia checkpoints to show downstream performance is predicted by pre-training loss, and redefines emergent abilities as those appearing below a loss threshold, persisting under continuous metrics.
Problem
Whether emergent abilities of language models are real tipping points tied to scale, given challenges that small models can match them and that emergence may be an artifact of discontinuous metrics.
Approach
Study emergence through pre-training loss rather than model size or compute: tasks such as MMLU, C-Eval, and GSM8K stay at random chance until pre-training loss falls below about 2.2 regardless of model size or token count, and this holds under continuous metrics. Emergence is redefined as an ability present at lower loss but absent at higher loss.
Key results
- Pre-training loss strongly predicts downstream performance: Spearman correlations around -0.98 to -0.996 on TriviaQA, HellaSwag, RACE, WinoGrande and similar tasks.
- On MMLU, C-Eval, GSM8K and GSM8K-Chinese, performance stays at random level until pre-training loss drops to about 2.2, after which it improves.
- Emergent improvements on MMLU persist under continuous metrics (CorrectChoiceProb and Brier score), indicating the tipping point is not an artifact of discontinuous accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Lower pre-training loss | Pretrained language models · TriviaQA, HellaSwag, RACE, WinoGrande, NLPCC-KBQA, ClozeT, CLUEWSC, and C3 | Task performance | Helped |
| Lower pre-training loss below about 2.2 | Pretrained language models · MMLU, C-Eval, GSM8K, and GSM8K-Chinese | Task performance: about 2.2 | Helped |
| Use pre-training loss rather than model size or token count to align model comparisons | Pretrained language models · 12 downstream tasks | Task performance at matched pre-training loss | No clear effect |
| Use pre-training loss rather than training compute as the performance indicator | 1.5B, 6B, and 32B models · Most evaluated downstream tasks | Alignment of performance trends across models | Helped |
| Evaluate emergent task gains with continuous metrics as well as accuracy | Pretrained language models · MMLU and C-Eval | Emergent performance improvement: 0.75 | No clear effect |
| Reduce pre-training loss below 2.2 | Pretrained language models · MMLU | Accuracy: 2.2; 25% | Helped |
Surprising findings
- LLaMA-65B underperforms smaller LLaMA models at the same loss early in training (loss above 1.8), an exception the authors attribute to possible smoothing in the plots.
- Compute-based curves do not collapse onto a single trend across model sizes, while loss-based curves do.
Limitations
- Only one architecture family and AdamW optimizer were studied; other architectures and optimizers are not covered.
- Pre-training loss depends on tokenizer and corpus, so loss values are not directly comparable across setups.
- LLaMA results were extracted from figures and intermediate checkpoints were unavailable; Pythia cannot analyze emergence on MMLU or GSM8K.
Experimental setup
Datasets: English and Chinese pre-training corpus (RedPajama-like pipeline) TriviaQA HellaSwag RACE WinoGrande MMLU GSM8K NLPCC-KBQA ClozeT CLUEWSC C3 C-Eval
Benchmarks: TriviaQA HellaSwag RACE WinoGrande MMLU GSM8K NLPCC-KBQA ClozeT CLUEWSC C3 C-Eval GSM8K-Chinese
Metrics: accuracy exact match Spearman correlation Pearson correlation CorrectChoiceProb Brier score
Baselines: LLaMA Pythia
Builds on: Chinchilla scaling laws Wei et al. emergent abilities Schaeffer et al. mirage analysis LLaMA Pythia
Closest papers by meaning
- Training on the Test Task Confounds Evaluation and Emergence (ICLR 2025 · similarity 0.91)
- U-shaped and Inverted-U Scaling behind Emergent Abilities of Large Language Models (ICLR 2025 · similarity 0.90)
- Predicting Emergent Abilities with Infinite Resolution Evaluation (ICLR 2024 · similarity 0.90)
- Observational Scaling Laws and the Predictability of Langauge Model Performance (NeurIPS 2024 · similarity 0.89)
- An Analysis for Reasoning Bias of Language Models with Small Initialization (ICML 2025 · similarity 0.88)
- Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.