Research map / Large language models / Language, safety and interpretability
Sloth: scaling laws for LLM skills to predict multi-benchmark performance across families
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sloth (Skills Scaling Laws), which fits a shared low-dimensional latent-skill scaling law on public leaderboard data across LLM families so that larger models' benchmark scores can be predicted without training them.
Problem
Scaling laws for benchmark performance vary across model families, and family-specific laws require training many models of increasing size; a single law generalizes poorly across families.
Approach
Model benchmark scores as linear combinations of a few latent skills (factor-analysis style) whose levels follow a translog function of log parameters and log tokens with family-specific intercepts (efficiency) and shared slopes, optionally with a learned monotone link function; fitted on public Open LLM Leaderboard data.
Key results
- Sloth prediction errors are similar to or lower than the 'Size and Tokens' variant across the 12 benchmarks under leave-one-family-out evaluation.
- A shared-intercept Sloth is a strong alternative to the FLOPs baseline on a subset of benchmarks.
- Sloth with item response theory predicts pass@k test-time scaling on MATH for held-out LLaMA 3 Instruct, Gemma and Pythia models.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use low-dimensional shared latent skills to model correlated benchmark scores | Sloth benchmark-performance prediction · Open LLM Leaderboard v1/v2 benchmarks | MAE | Helped |
| Instruction-tune the base model | LLM latent skills · Instruction Following | skill level | Helped |
| Instruction-tune the base model | LLM latent skills · Reasoning | skill level | Mixed |
| Instruction-tune the base model | LLM latent skills · Knowledge | skill level | No clear effect |
Surprising findings
- Instruction tuning can either increase or decrease reasoning skill, with no clear rule across families.
Limitations
- Still requires observing at least one model from the target family for good predictions.
- Neural link function may interpolate well but lacks extrapolation guarantees.
- Identifiability theory covers only the simple fixed-sigmoid version of the model.
Experimental setup
Datasets: Open LLM Leaderboard v1 Open LLM Leaderboard v2 HumanEval EQ-Bench AgentBench MATH
Benchmarks: GSM8K MATH Lvl 5 MMLU MMLU-PRO BBH GPQA MUSR TruthfulQA HellaSwag Winogrande ARC IFEval
Metrics: mean absolute error (percentage points) mean absolute percentage error pass@k
Baselines: FLOPs (shared intercept) FLOPs PCA + FLOPs Size and Tokens
Builds on: Chinchilla scaling laws Observational scaling laws translog production functions factor analysis
Closest papers by meaning
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.90)
- Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.90)
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.90)
- Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs (ICLR 2026 · similarity 0.90)
- LLMs on the Line: Data Determines Loss-to-Loss Scaling Laws (ICML 2025 · similarity 0.90)
- Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for LLM Problem-Solving (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.