Research map / Large language models / Language, safety and interpretability
U-shaped and Inverted-U Scaling behind Emergent Abilities of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that grouping questions by difficulty reveals U-shaped scaling on hard questions and inverted-U then steady-improvement scaling on easy questions, which together explain emergent abilities, and proposes Slice-and-Sandwich to forecast emergence from pre-threshold models.
Problem
Emergent abilities in LLMs appear as stagnant-then-sharp performance jumps that are hard to predict from smaller models; the mechanism behind them and how to forecast the emergence threshold and post-threshold performance remain unclear.
Approach
Group questions by difficulty (measured with a Target-Conditioned Brier score over sub-threshold models); easy questions show inverted-U then steady scaling (deep double descent) and hard questions show U-shaped scaling, whose offsetting trends produce stagnation. Slice-and-Sandwich fits each group's trend with polynomials on pre-threshold models, averages them, and maps the result back to accurac
Key results
- On MMLU, arithmetic and Persian-QA, hard question groups show U-shaped scaling while easy groups show inverted-U followed by steady improvement, with the reversion roughly coinciding with the emergence threshold.
- TC Brier Score correlates with accuracy at Pearson 0.99 on MMLU, 1.00 on arithmetic and 0.88 on Persian-QA, higher than the un-conditionalized Brier variant.
- Slice-and-Sandwich forecasts the post-threshold soaring trend of MMLU and arithmetic accuracy using only models below the threshold, while the Sigmoid baseline does not capture MMLU's soar.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Condition probabilities on available classes rather than using the un-conditionalized TC Brier Score (ablation) | TC Brier Score · MMLU | Pearson correlation with accuracy: 0.96 -> 0.99 | Helped |
| Condition probabilities on available classes rather than using the un-conditionalized TC Brier Score (ablation) | TC Brier Score · MMLU | Spearman correlation with accuracy: 0.87 -> 0.91 | Helped |
| Condition probabilities on available classes rather than using the un-conditionalized TC Brier Score (ablation) | TC Brier Score · MMLU | Kendall correlation with accuracy: 0.73 -> 0.79 | Helped |
| Condition probabilities on available classes rather than using the un-conditionalized TC Brier Score (ablation) | TC Brier Score · Arithmetic | Pearson correlation with accuracy: 0.93 -> 1.00 | Helped |
| Condition probabilities on available classes rather than using the un-conditionalized TC Brier Score (ablation) | TC Brier Score · Arithmetic | Spearman correlation with accuracy: 0.93 -> 0.97 | Helped |
| Condition probabilities on available classes rather than using the un-conditionalized TC Brier Score (ablation) | TC Brier Score · Arithmetic | Kendall correlation with accuracy: 0.79 -> 0.88 | Helped |
| Condition probabilities on available classes rather than using the un-conditionalized TC Brier Score (ablation) | TC Brier Score · Persian-QA | Pearson correlation with accuracy: 0.60 -> 0.88 | Helped |
| Condition probabilities on available classes rather than using the un-conditionalized TC Brier Score (ablation) | TC Brier Score · Persian-QA | Spearman correlation with accuracy: 0.52 -> 0.68 | Helped |
| Condition probabilities on available classes rather than using the un-conditionalized TC Brier Score (ablation) | TC Brier Score · Persian-QA | Kendall correlation with accuracy: 0.37 -> 0.52 | Helped |
Surprising findings
- Non-emergent tasks (ARC, HellaSwag, abstract narrative understanding) do not show the U-shaped or inverted-U patterns.
- Under accuracy rather than TC Brier Score, the U-shaped scaling of hard questions is mitigated and hard groups stagnate near zero accuracy.
Limitations
- Focused on multiple-choice tasks; applying the method to string-matching tasks requires a suitable continuous metric.
- Polynomial-degree and train-test split choices affect forecasts; higher-degree fits and lower thresholds are less robust.
- Emergence threshold is identified manually and the approach relies on prior belief about polynomial functional forms.
Experimental setup
Datasets: MMLU BIG-bench arithmetic BIG-bench Persian-QA BIG-bench Hindu knowledge BIG-bench conceptual combinations BIG-bench analogical similarity ARC HellaSwag
Benchmarks: MMLU BIG-bench arithmetic BIG-bench Persian-QA BIG-bench Hindu knowledge BIG-bench conceptual combinations BIG-bench analogical similarity BIG-bench abstract narrative understanding ARC HellaSwag
Metrics: accuracy TC Brier Score Pearson correlation Spearman correlation Kendall correlation
Baselines: Sigmoid-based regression scaling law Hard-Lift
Builds on: deep double descent Sigmoid-based task-specific scaling law
Closest papers by meaning
- Predicting Emergent Abilities with Infinite Resolution Evaluation (ICLR 2024 · similarity 0.91)
- Understanding Emergent Abilities of Language Models from the Loss Perspective (NeurIPS 2024 · similarity 0.90)
- Observational Scaling Laws and the Predictability of Langauge Model Performance (NeurIPS 2024 · similarity 0.88)
- Provable Scaling Laws for the Test-Time Compute of Large Language Models (NeurIPS 2025 · similarity 0.88)
- Training on the Test Task Confounds Evaluation and Emergence (ICLR 2025 · similarity 0.87)
- Prescriptive Scaling Reveals the Evolution of Language Model Capabilities (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.