Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Bayesian Neural Scaling Law Extrapolation with Prior-Data Fitted Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NSL-PFN, a prior-data fitted network meta-trained on synthetic scaling curves (power-law, breaks, upward BetaCDF segments) to give Bayesian, uncertainty-aware neural scaling law extrapolation.
Problem
Neural scaling law extrapolation mostly uses point estimates without uncertainty, and existing Bayesian MCMC methods struggle with non-monotonic, chaotic curves and need slow sampling.
Approach
Meta-train a transformer PFN on a tailored synthetic prior that samples downward power-law segments (M3/M4), random breaks, and upward BetaCDF segments, so one forward pass yields a posterior predictive distribution over future scaling behavior.
Key results
- Average over benchmarks: NSL-PFN RMSLE 0.0280 and LL 3.330 on IC, and RMSLE 0.0194 and LL 2.773 on NLP/Nano, best among compared methods.
- On double descent (DD), NSL-PFN achieves RMSLE 0.0335 and LL 2.565 versus 0.0494/1.250 for MCMC (BNSL) and 0.0706/1.321 for LC-PFN.
- Inference takes about 0.02 s per curve for NSL-PFN versus 15.65 to 280.55 s for point estimation and MCMC baselines (nsamples=150).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NSL-PFN instead of the strongest baseline | neural scaling law extrapolation · IC average | RMSLE: 0.0406 -> 0.0280 | Helped |
| Use NSL-PFN instead of the strongest baseline | neural scaling law extrapolation · IC average | LL: 3.024 -> 3.330 | Helped |
| Use NSL-PFN instead of the strongest baseline | neural scaling law extrapolation · NLP and Nano average | RMSLE: 0.0223 -> 0.0194 | Helped |
| Use NSL-PFN instead of the strongest baseline | neural scaling law extrapolation · NLP and Nano average | LL: 2.531 -> 2.773 | Helped |
| Use NSL-PFN instead of the strongest baseline | neural scaling law extrapolation · ColPret | RMSLE: 0.0289 -> 0.0271 | Helped |
| Use NSL-PFN instead of the strongest baseline | neural scaling law extrapolation · ColPret | LL: 2.804 -> 2.794 | No clear effect |
| Use NSL-PFN instead of the strongest baseline | neural scaling law extrapolation · DD | RMSLE: 0.0468 -> 0.0335 | Helped |
| Use NSL-PFN instead of the strongest baseline | neural scaling law extrapolation · DD | LL: 1.321 -> 2.565 | Helped |
| Include upward segments in the prior (ablation) | NSL-PFN · IC and ColPret | RMSLE: IC: 0.032 -> 0.028; ColPret: 0.032 -> 0.027 | Helped |
| Include upward segments in the prior (ablation) | NSL-PFN · DD | RMSLE: 0.051 -> 0.033 | Helped |
| Include upward segments in the prior (ablation) | NSL-PFN · NLP and Nano average | RMSLE: 0.015 -> 0.019 | Hurt |
| Allow random breaks in the prior (ablation) | NSL-PFN · IC, NLP and Nano, ColPret, and DD | RMSLE: IC: 0.038 -> 0.032; NLP and Nano: 0.015 -> 0.015; ColPret: 0.032 -> 0.027; DD: 0.071 -> 0.051 | Helped |
Surprising findings
- On NMT and LM, the simpler point estimator MCMC (M4) and M4 outperform NSL-PFN, which the authors attribute to the prior's complex shapes acting as a distractor on simple curves.
- Increasing MCMC sample counts gives marginal gains and for MCMC (M1) and MCMC (BNSL) performance slightly degrades with longer chains.
Limitations
- Performs worse than simpler models (M4, MCMC M4) on simple scaling curves such as NMT and LM because the prior includes complex behaviors.
- Cannot predict unexpected future breaks when the context ends in a single decreasing segment (failure case on DD).
- Prior hyperparameters were set manually to match visual curve shapes; Bayesian optimization gave only slight gains.
Experimental setup
Datasets: scaling laws benchmark (IC, NLP: NMT, LM, BB) nanoGPT-Bench (Nano) ColPret double descent (DD) scaling laws synthetic prior samples
Benchmarks: IC (ImageNet, CIFAR100, Birds, Caltech101) NLP (NMT, LM, BB) Nano ColPret DD
Metrics: RMSLE average log-likelihood mean squared calibration error Bayesian active learning performance
Baselines: M1 M2 M3 M4 BNSL MCMC variants (EMCEE) LC-PFN BLR with NN basis DKGP
Builds on: Prior-data Fitted Networks LC-PFN Broken Neural Scaling Laws Revisiting neural scaling laws (M1-M4) EMCEE
Closest papers by meaning
- Position: The Future of Bayesian Prediction Is Prior-Fitted (ICML 2025 · similarity 0.87)
- A Symmetry-Aware Exploration of Bayesian Neural Network Posteriors (ICLR 2024 · similarity 0.87)
- Critical feature learning in deep neural networks (ICML 2024 · similarity 0.87)
- Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra (ICLR 2025 · similarity 0.87)
- A Study of Bayesian Neural Network Surrogates for Bayesian Optimization (ICLR 2024 · similarity 0.86)
- Scaling Laws in Linear Regression: Compute, Parameters, and Data (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.