Research map / Large language models / Language, safety and interpretability
Are Emergent Abilities of Large Language Models a Mirage?
Rylan Schaeffer, Brando Miranda, Sanmi Koyejo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Accuracy with Token Edit Distance while keeping model outputs fixed | InstructGPT/GPT-3 · 2-shot 2-digit multiplication and 2-shot 4-digit addition | performance versus model scale | Improved |
| Increase test-data size when measuring Accuracy | InstructGPT/GPT-3 · 2-shot 2-digit multiplication and 2-shot 4-digit addition | accuracy | Improved |
| Measure longer target strings with Token Edit Distance | InstructGPT/GPT-3 · arithmetic tasks, target strings of length 1 to 5 | Token Edit Distance | Hurt |
| Measure longer target strings with Accuracy | InstructGPT/GPT-3 · arithmetic tasks | accuracy | Hurt |
| Replace Multiple Choice Grade with Brier Score | LaMDA · BIG-Bench tasks where emergence was reported | emergence under the evaluation metric | Hurt |
| Use discontinuous Reconstruction_c instead of squared reconstruction error | Shallow nonlinear autoencoders · CIFAR100 natural-image reconstruction | apparent emergence with scale | Improved |
| Score whether all images in a sequence are classified correctly | Autoregressive Transformers · Omniglot classification | subset accuracy | Improved |
| Score whether all K test examples are classified correctly instead of using test accuracy | LeNet · MNIST | subset accuracy | Improved |
Closest papers by meaning
- Understanding Emergent Abilities of Language Models from the Loss Perspective (NeurIPS 2024 · similarity 0.93)
- Predicting Emergent Abilities with Infinite Resolution Evaluation (ICLR 2024 · similarity 0.91)
- U-shaped and Inverted-U Scaling behind Emergent Abilities of Large Language Models (ICLR 2025 · similarity 0.89)
- Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space (NeurIPS 2024 · similarity 0.89)
- Training on the Test Task Confounds Evaluation and Emergence (ICLR 2025 · similarity 0.89)
- Observational Scaling Laws and the Predictability of Langauge Model Performance (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
