Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Zero-Shot Performance Prediction for Probabilistic Scaling Laws
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Frames small-scale NLP learning-curve prediction as bi-level hierarchical multitask learning with latent-variable multi-output GPs, enabling zero-shot curve prediction, probabilistic scaling laws via Monte Carlo, and active-learning queries.
Problem
Predicting learning curves and deriving scaling laws for NLP models is computationally expensive because many models must be trained; existing zero-shot curve predictors need large datasets and ignore hierarchical structure in NLP learning-curve data.
Approach
Model each NLP learning-curve dataset as a two-layer hierarchy (tasks and data instances) and use a latent variable multi-output GP (Ma-GP) to capture hierarchy and cross-task correlations, enabling zero-shot curve prediction, probabilistic scaling laws via Monte Carlo, and variance-based active learning of which curves to train.
Key results
- On nanoGPT-large zero-shot last-point prediction, Ma-GP achieves MSE 0.01 (Quad), 0.00 (Tri), and 0.00 (T1), versus DHGP 0.03/0.02/0.01 and BNN variants around 14-24.
- Active learning over queries yields AbC values consistently below the initial training set and more certain across runs than random, smallest-first, or largest-first strategies.
- Ma-GP outperforms NBL and DHGP in RMSE on the bilingual translation dataset and outperforms NRBL and baselines in RMSE for few-shot M2M100-1.2B extrapolation, though with higher predictive uncertainty.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MaGP instead of DHGP | nanoGPT large Quad last-point prediction · nanoGPT large Quad, last extrapolated point | MSE: 0.03±0.00 -> 0.01±0.01 | Helped |
| Use MaGP instead of DHGP | nanoGPT large Quad last-point prediction · nanoGPT large Quad, last extrapolated point | MAE: 0.17±0.00 -> 0.07±0.03 | Helped |
| Use MaGP instead of DHGP | nanoGPT large Quad last-point prediction · nanoGPT large Quad, last extrapolated point | MNLPD: 0.51±0.08 -> -0.93±1.43 | Helped |
| Use MaGP instead of DHGP | nanoGPT large Tri last-point prediction · nanoGPT large Tri, last extrapolated point | MSE: 0.02±0.00 -> 0.00±0.00 | Helped |
| Use MaGP instead of DHGP | nanoGPT large Tri last-point prediction · nanoGPT large Tri, last extrapolated point | MAE: 0.15±0.00 -> 0.06±0.02 | Helped |
| Use MaGP instead of DHGP | nanoGPT large Tri last-point prediction · nanoGPT large Tri, last extrapolated point | MNLPD: -0.22±0.13 -> -1.27±0.21 | Helped |
| Use MaGP instead of DHGP | nanoGPT large T1 last-point prediction · nanoGPT large T1, last extrapolated point | MSE: 0.01±0.00 -> 0.00±0.01 | Helped |
| Use MaGP instead of DHGP | nanoGPT large T1 last-point prediction · nanoGPT large T1, last extrapolated point | MAE: 0.12±0.00 -> 0.05±0.05 | Helped |
| Use MaGP instead of DHGP | nanoGPT large T1 last-point prediction · nanoGPT large T1, last extrapolated point | MNLPD: -0.67±0.02 -> -1.08±0.92 | Helped |
| Use MaGP instead of DHGP | nanoGPT small T1 last-point prediction · nanoGPT small T1, last extrapolated point | MSE: 0.04±0.01 -> 0.05±0.00 | Hurt |
| Use MaGP instead of DHGP | nanoGPT small T1 last-point prediction · nanoGPT small T1, last extrapolated point | MAE: 0.19±0.00 -> 0.22±0.00 | Hurt |
| Use MaGP instead of DHGP | nanoGPT small T1 last-point prediction · nanoGPT small T1, last extrapolated point | MNLPD: -0.23±0.04 -> -0.03±0.01 | Hurt |
Surprising findings
- On the small nanoGPT-small T1 split, DHGP outperforms Ma-GP on last-point MSE and MAE (0.04 vs 0.05).
Limitations
- Only 11 sampled points per learning curve and a five-by-six grid of curves due to computational constraints.
- Cannot interpolate to unseen embedding-parameter values; unseen layer counts yield averaged estimates.
- Active learning ignores query compute cost; only variance-based uncertainty reduction is used.
Experimental setup
Datasets: FineWeb-Edu (10B) EMNLP2021 large-scale multilingual translation small track (OPUS) Flores101
Benchmarks: nanoGPT learning-curve datasets (nanoGPT-large, nanoGPT-small) bilingual translation learning curves multilingual M2M100 learning curves
Metrics: MSE MAE MNLPD RMSE Area between Curves (AbC)
Baselines: DHGP BNN (orig) BNN (LC) naive baseline (NBL) naive regression baseline (NRBL)
Builds on: latent variable multi-output Gaussian processes deep hierarchical Gaussian processes Bayesian neural networks for learning curve extrapolation
Closest papers by meaning
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.88)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.87)
- Resolving Discrepancies in Compute-Optimal Scaling of Language Models (NeurIPS 2024 · similarity 0.87)
- Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules (NeurIPS 2025 · similarity 0.87)
- Scaling Inference-Efficient Language Models (ICML 2025 · similarity 0.86)
- Prescriptive Scaling Reveals the Evolution of Language Model Capabilities (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.