Research map / Architectures and efficiency / RNNs and normalization
Navigating Scaling Laws: Compute Optimality in Adaptive Model Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes adaptive-shape training that switches ViT patch size, LM context length, width, batch size, and training objective mid-run at points chosen by fitted scaling laws, cutting FLOPs by up to ~60% for ViTs.
Problem
Static training keeps model shape fixed, but compute-optimal shape varies with compute budget; the paper asks how to schedule shape changes during training to reach a target performance with fewer FLOPs.
Approach
Fit power-law scaling laws per shape configuration, invert them, and switch shape at the transition points where the derivative of inverse-law (compute needed per unit error reduction) is maximized; FlexiViT-style patch-size resizing and random-init width expansion make the switches smooth.
Key results
- Patch-size scheduling allows up to -60% FLOPs to reach the same ImageNet-1k 10-shot error for V640-12, and halves the compute needed to optimally train ViT-Base.
- Context-size scheduling for Llama-style LMs reduces FLOPs by up to 40%.
- Scheduled training outperforms FlexiViT's random patch-size scheduler and linear/logarithmic transition schedules.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Schedule patch size using scaling-law transition points instead of keeping patch size fixed | Vision Transformer training · ImageNet-1 10-shot | training FLOPs to reach the same performance: -60% FLOPs | Helped |
| Use the scaling-law schedule instead of uniform random patch-size sampling | Vision Transformer training · ImageNet-1 10-shot | training FLOPs to reach the same performance | Helped |
| Use the scaling-law schedule instead of linear or logarithmic patch-size transitions | Vision Transformer training · ImageNet-1 10-shot | training FLOPs to reach the same performance | Helped |
| Schedule context size using scaling-law transition points rather than keeping it fixed | Llama language-model training · English Wikipedia and Books validation | training FLOPs to reach the same performance: up to 40% | Helped |
| Schedule model width at scaling-law transition points rather than keeping width fixed | Vision Transformer training · ImageNet-1 10-shot | training FLOPs to reach the same performance | Helped |
| Schedule batch size by transitioning from smaller to larger batches | V384-20/12 training · ImageNet-1 10-shot | training FLOPs to reach the same performance | Helped |
| Switch from supervised training to distillation at the scaling-law transition point | V384-20/12 training · ImageNet-1 10-shot | training FLOPs to reach the same performance: fewer FLOPs | Helped |
| Schedule both model width and patch size instead of adapting only one | Vision Transformer training · ImageNet-1 10-shot | training FLOPs to reach the same performance | Helped |
| Reduce patch size to 6 (ablation) | V256-6 training · ImageNet-1 10-shot | performance: some absolute gains | Mixed |
| Reduce patch size to 4 (ablation) | V256-6 training · ImageNet-1 10-shot | performance | Hurt |
| Initialize expanded width weights with zero values instead of random weights scaled to existing weight norms (ablation) | Vision Transformer width expansion · ImageNet-1 10-shot | performance | Hurt |
| Change body and head weight decay values (ablation) | Vision Transformer training · ImageNet-1 | performance | No clear effect |
Surprising findings
- Reducing patch size below 4 (to 4) causes performance to worsen, so the patch-size vs performance trend is not monotonic.
- Zero-initializing new weights during width expansion (a function-preserving scheme) performed suboptimally compared with small random initialization.
Limitations
- Requires fitting scaling laws per shape parameter, which is expensive to obtain.
- Learning rate is not scheduled and no cooldown phase is used, so conclusions may shift with LR schedules.
- Gains are largest early in training; not expected to hold for infinite compute.
Experimental setup
Datasets: ImageNet-21k ImageNet-1k English Wikipedia 20220301.en bookcorpus
Benchmarks: ImageNet-1k 10-shot ImageNet-1k fine-tuning Pets Birds
Metrics: 10-shot top-1 error validation cross-entropy loss FLOPs to reach target performance
Baselines: FlexiViT fixed patch size ViT linear patch size schedule logarithmic patch size schedule
Builds on: FlexiViT Chinchilla scaling laws Transformer++ (Llama recipe)
Closest papers by meaning
- A Dynamical Model of Neural Scaling Laws (ICML 2024 · similarity 0.91)
- 4+3 Phases of Compute-Optimal Neural Scaling Laws (NeurIPS 2024 · similarity 0.89)
- Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations (NeurIPS 2024 · similarity 0.89)
- Scaling Laws in Linear Regression: Compute, Parameters, and Data (NeurIPS 2024 · similarity 0.88)
- Resolving Discrepancies in Compute-Optimal Scaling of Language Models (NeurIPS 2024 · similarity 0.87)
- Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.