Research map / Optimization / Implicit bias and generalization
Functional Scaling Laws in Kernel Regression: Loss Dynamics and Learning Rate Schedules
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Analyzes SGD on power-law kernel regression via an intrinsic-time SDE and derives a Functional Scaling Law (FSL) describing full loss trajectories under arbitrary learning-rate schedules, with explicit scaling results for constant, exponential-decay, and WSD schedules.
Problem
Existing scaling laws focus on final-step loss and do not explain how the full loss dynamics and the learning rate schedule (LRS) shape scaling efficiency in pre-training.
Approach
Introduces intrinsic time (accumulated learning rate) and an intrinsic-time SDE in which the LRS enters only through the noise term, yielding a Volterra/convolutional Functional Scaling Law that captures the whole loss trajectory and explains why WSD and learning-rate decay help.
Key results
- FSL closely matches averaged SGD loss trajectories under cosine, WSD-like and cyclic schedules on power-law kernel regression (s=0.5, beta=4).
- Scaling analysis: WSD gives the best data- and compute-optimal scaling, followed by exponential decay, then constant LRS (e.g., hard-regime data-optimal excess risk D^{-s} for WSD vs D^{-s/(s+1)} for constant).
- FSL fitted on 8-1-1 LRS trajectories predicts cosine and WSD loss curves for 400M and 1B Llama models; the FSL-optimal LRS is WSD-like and gives a lower final loss than baselines on a 1B QwenMoE.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use WSD-like learning-rate decay instead of exponential decay or a constant schedule | SGD in power-law kernel regression · PLK regression | final-step excess risk | Helped |
| Use the FSL-optimal learning-rate schedule instead of cosine, WSD, or 8-1-1 | 1B QwenMoE · 20B tokens | final loss | Helped |
| Use the FSL-optimal learning-rate schedule instead of WSD schedules with stable-to-decay ratios 3:1, 4:1, or 5:1 (ablation) | 400M LLaMA · 20B tokens | final loss: approximately 0.01 | Helped |
| Reduce the final learning rate to near zero instead of using a final rate of 1/10 of the peak (ablation) | 100M LLaMA · 40B tokens | final loss | No clear effect |
| Vary the WSD stable-to-decay ratio among 3:1, 4:1, and 5:1 (ablation) | 400M LLaMA · 20B tokens | final loss | No clear effect |
Surprising findings
- Even quadratic optimization (kernel regression) reproduces the advantage of WSD schedules over pure decay.
- The FSL-optimal schedule decays the learning rate far below the conventional 0.1*peak threshold; decaying to near zero gave no significant further loss reduction in experiments.
Limitations
- Theory is for power-law kernel regression with one-pass SGD and continuous-time approximation, not deep networks.
- Random-feature results only proven for s<=1; easy tasks with s>1 left open.
- LLM experiments are limited to 0.1B-1B models and fitting relies on an empirical ansatz with several free parameters.
Experimental setup
Datasets: synthetic power-law kernel regression data LLM pre-training corpus (20B tokens)
Benchmarks: SGD loss trajectories on power-law kernel regression LLM pre-training validation loss curves
Metrics: excess risk fit and prediction accuracy of loss curves final-step loss
Baselines: Chinchilla law Momentum Law Multi-Power Law cosine schedule WSD schedule 8-1-1 schedule
Builds on: Chinchilla scaling law Multi-Power Law Momentum Law power-law random features / kernel scaling analysis
Closest papers by meaning
- Scaling Law with Learning Rate Annealing (NeurIPS 2025 · similarity 0.92)
- Resolving Discrepancies in Compute-Optimal Scaling of Language Models (NeurIPS 2024 · similarity 0.91)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.91)
- A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules (ICLR 2025 · similarity 0.91)
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.90)
- Universal One-third Time Scaling in Learning Peaked Distributions (ICML 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.