Research map / Architectures and efficiency / RNNs and normalization
Active Budget Allocation for Efficient Scaling Law Estimation via Surrogate-Guided Pruning
Viktoria Schram, Markus Hiller, Daniel Beck, Trevor Cohn
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add LMC surrogate-guided pruning to Successive Halving | Successive Halving · synthetic learning curves, 5 models, B=10^4 petaFLOPs | mean relative improvement in minimum loss: +5.47 % | Improved |
| use more candidate models (5 to 20) when adding LMC surrogate to SH (ablation) | SH LMC · synthetic learning curves, B=10^4 | mean relative improvement over SH: 5.47 % -> 1.50 % | Hurt |
| use Successive Halving instead of uniform allocation (ablation) | budget allocation · synthetic learning curves | relative loss: UA -10.17 % (5 models, B=10^2) vs SH | Improved |
| use LMC surrogate instead of Deep Ensemble surrogates in SH (ablation) | SH on nanoGPT · nanoGPT dataset, 20 models, B=10^4 | mean relative improvement over SH: 2.84 % vs 2.02 % (PL), 1.35 % (EXP), 1.80 % (MMF) | Improved |
| add LMC surrogate to SH on real-world nanoGPT curves | Successive Halving · nanoGPT dataset | mean relative improvement over SH: up to 2.84 % | Improved |
| add LMC surrogate to SH, scaling law accuracy | Successive Halving · nanoGPT, AbC to Full Data SL | area between curves: 0.12 -> 0.11 (20 models, B=10^4); 0.09 -> 0.11 (5 models, B=10^4) | No clear effect |
| add LMC surrogate to SH, loss regret | Successive Halving · nanoGPT, 20 models, B=10^4 | loss regret: 0.67 -> 0.59 | Improved |
| add SH-based allocation versus exhaustive training of all models | scaling law estimation · nanoGPT, 20 models, B=10^4 | compute cost saving: saving 98.70% | Improved |
| add GP mean extrapolation after SH LMC allocation (ablation) | SH LMC · Kaplan synthetic LCs, B=10^5 | AbC to ground truth SL: 2.17 -> 0.00 ± 0.00 | Improved |
| fit scaling law to GP UCB instead of GP mean (ablation) | GP extrapolation · Kaplan synthetic LCs, B=10^5 | AbC to ground truth SL: 0.00 -> 0.53 | Hurt |
| add noise (white, Ornstein-Uhlenbeck, Brownian) to learning curves (ablation) | SH LMC vs SH · synthetic learning curves, 5 and 20 models | average minimum loss | No clear effect |
Closest papers by meaning
- Power Lines: Scaling laws for weight decay and batch size in LLM pre-training (NeurIPS 2025 · similarity 0.87)
- Kinetics: Rethinking Test-Time Scaling Law (NeurIPS 2025 · similarity 0.87)
- Unified Scaling Laws for Compressed Representations (NeurIPS 2025 · similarity 0.86)
- Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations (NeurIPS 2024 · similarity 0.86)
- Scaling laws for learning with real and surrogate data (NeurIPS 2024 · similarity 0.86)
- Asymmetric Scaling Laws from Sparse Features (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
