Research map / Architectures and efficiency / RNNs and normalization

Neural Neural Scaling Laws

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Neural scaling laws · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes NeuNeu, a small transformer with a conv loss encoder that forecasts downstream task accuracy from token-level validation probabilities and accuracy trajectories, trained on DataDecide checkpoints to beat logistic scaling laws.

Problem

Predicting downstream task performance of language models from pretraining validation loss is limited by averaging away token-level loss information and by restrictive parametric scaling-law forms that cannot capture diverse per-task scaling behaviors.

Approach

Frame scaling-law prediction as time-series extrapolation with a learned neural predictor that ingests token-level validation probabilities (via a 1D CNN encoder) plus observed accuracy and compute-gap sequences, outputting quantile predictions; inputs are made scale-invariant via relative compute gaps and loss-to-probability conversion.

Key results

• NeuNeu achieves 1.99% MAE across 66 downstream tasks, a 44% reduction relative to logistic scaling laws (3.56% MAE).
• NeuNeu ranks competing training runs correctly 76.6% of the time versus 63.7% for logistic scaling laws (12.9% improvement).
• The 10%-90% predicted interquantile interval captures 77.6% of ground-truth accuracies against an 80% target.

Surprising findings

• Parametric scaling laws (logistic, BNSL) show roughly 4x higher error on Pythia and OLMo-Hybrid than on in-distribution seeds.
• NeuNeu predicts unseen held-out tasks with lower MAE than logistic laws fit directly to those tasks.

Limitations

• Evaluated only on classification-style tasks where accuracy is the metric; generative tasks may scale differently.
• Training limited to today's open-source checkpoints, mostly transformer runs from DataDecide.
• Logistic and BNSL baselines are given ground-truth average loss, a strict advantage that is not available in practice.

Experimental setup

Datasets: DataDecide WebOrganizer OLMES DCLM-Baseline C4 Pythia runs OLMo-Hybrid-7B runs

Benchmarks: OLMES tasks (66 DataDecide tasks) DCLM-Baseline seed 2 generalization C4 generalization Pythia generalization OLMo-Hybrid-7B generalization

Metrics: mean absolute error interquantile interval coverage pairwise ranking accuracy

Baselines: Logistic scaling law LC-PFN BNSL NoLoss Average

Builds on: logistic scaling laws broken neural scaling laws LC-PFN DataDecide BERT RoPE

Closest papers by meaning

• Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.92)
• On the Invariance and Generality of Neural Scaling Laws (NeurIPS 2026 · similarity 0.91)
• A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.91)
• A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.90)
• A Simple Model of Inference Scaling Laws (ICML 2025 · similarity 0.89)
• Scaling Law with Learning Rate Annealing (NeurIPS 2025 · similarity 0.89)