Research map / Architectures and efficiency / RNNs and normalization

On the Invariance and Generality of Neural Scaling Laws

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Neural scaling laws · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes an information-resolution scaling law that is invariant under bijective data transformations and modified by a variance-inflation and loss-shift term under non-bijective ones, enabling cross-domain scaling prediction for language, vision, speech and clinical time series.

Problem

Fitting neural scaling laws for a new model-task pair requires expensive sweeps; existing data-dependent scaling laws do not explain when and why scaling exponents change under data transformations and predict a quality-independent loss floor.

Approach

Characterizes data transformations by information resolution rho(T)=I(X';Y)/I(X;Y): bijective transforms leave the Chinchilla law unchanged, while non-bijective ones add a variance-inflation factor rho^-nu on the data term and an optimal-loss-shift term kappa(1-rho)^mu to the irreducible loss.

Key results

• Cross-domain predictions recover the data-scaling exponent beta within 3% relative error for MIMIC-IV noise injection and OpenWebText-to-clinical transfer, versus larger errors for Quality-Aware and Sub-Optimal baselines.
• Table 2 reports the mortality noise-injection setting at rho=0.25 with Real E=5.1371, Ours E=4.7839, while Quality-Aware and Sub-Optimal predict E near 2.2-2.4.
• Optimal model size shrinks as a power law in rho, N*(C,rho)=N*(C,1)*rho^(nu/(alpha+beta)), validated empirically on WikiText-103, OpenWebText and MIMIC-IV.

Surprising findings

• Quantization reduces NLL while perplexity increases, because it shrinks the effective vocabulary.
• Vision and speech tasks tolerate substantially more information loss than language modeling before scaling visibly degrades.

Limitations

• Transfer of kappa, nu, mu across corpora assumes comparable task-relevant mutual information between source and target; weakens for unrelated domains such as source code.
• rho(T) must be estimated for implicit cross-corpus shifts, and the compression and n-gram estimators only approximate the true value.
• Framework assumes a single shared modality; cross-modality prediction lacks a natural transformation.

Experimental setup

Datasets: WikiText-103 OpenWebText C4 ImageNet-1K CIFAR-100 LibriSpeech MIMIC-IV MIMIC-IV discharge summaries

Benchmarks: WikiText-103 OpenWebText C4 ImageNet-1K CIFAR-100 LibriSpeech MIMIC-IV

Metrics: negative log-likelihood CTC loss scaling exponent beta error irreducible loss E

Baselines: Quality-Aware scaling law Sub-Optimal scaling law

Builds on: Chinchilla scaling law data-dependent scaling laws (quality-aware) sub-optimal scaling laws

Closest papers by meaning

• A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.92)
• Neural Neural Scaling Laws (NeurIPS 2026 · similarity 0.91)
• A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.91)
• Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.90)
• A Simple Model of Inference Scaling Laws (ICML 2025 · similarity 0.90)
• Superposition Yields Robust Neural Scaling (NeurIPS 2025 · similarity 0.90)