Research map / Architectures and efficiency / RNNs and normalization
Gemstones: A Model Suite for Multi-Faceted Scaling Laws
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Releases Gemstones, a suite of over 4000 checkpoints from transformers up to 2B parameters with diverse width/depth shapes and learning-rate/cooldown ablations, and uses it to show scaling-law prescriptions are sensitive to model selection and fitting choices.
Problem
Scaling laws are usually fit on a narrow family of models with frozen hyperparameters and a narrow range of width-depth ratios, so their prescriptions may not generalize and may depend heavily on design and model-selection choices.
Approach
Train a large open model suite varying width, depth, learning rate and cooldown, then fit scaling laws with a new convex-hull (lower envelope) fitting method that excludes suboptimal models, and quantify how fitting choices change prescriptions and how width/depth affects FLOPs versus GPU-hour efficiency.
Key results
- Scaling-law slope fitted on Gemstones varies widely with fitting choices, e.g. Approach 1 slope ranges from 0.4579 (all data) to 0.7987 (only tokens >120b), a delta of 0.34.
- Deeper models reach lower loss per FLOP and, at matched FLOPs, show increasing benchmark accuracy with depth, whereas wider models are more efficient in GPU hours.
- Skinny models waste over 200% of GPU hours relative to the best configuration while overspending only about 50% in FLOPs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use only checkpoints at tokens > 120b when fitting (ablation) | Gemstones Approach 1 scaling law · Gemstones | slope: 0.4579 -> 0.7987; 0.5689 -> 0.9666 | Mixed |
| Use only checkpoints at tokens ≤ 100b when fitting (ablation) | Gemstones Approach 1 scaling law · Gemstones | slope: 0.4579 -> 0.4994; 0.5689 -> 0.6269 | Mixed |
| Use cooldown runs when fitting (ablation) | Gemstones Approach 1 scaling law · Gemstones | slope: 0.4579 -> 0.5970; 0.5689 -> 0.7242 | Mixed |
| Use half-learning-rate runs when fitting (ablation) | Gemstones Approach 1 scaling law · Gemstones | slope: 0.4579 -> 0.5131; 0.5689 -> 0.6224 | Mixed |
| Use only checkpoints at tokens > 120b when fitting (ablation) | Gemstones Approach 3 scaling law · Gemstones | slope: 0.6965 -> 0.7515; 0.7141 -> 0.7350 | Mixed |
| Use only checkpoints at tokens ≤ 100b when fitting (ablation) | Gemstones Approach 3 scaling law · Gemstones | slope: 0.6965 -> 0.6986; 0.7141 -> 0.7030 | Mixed |
| Use cooldown runs when fitting (ablation) | Gemstones Approach 3 scaling law · Gemstones | slope: 0.6965 -> 0.6992; 0.7141 -> 0.7104 | Mixed |
| Use half-learning-rate runs when fitting (ablation) | Gemstones Approach 3 scaling law · Gemstones | slope: 0.6965 -> 0.6740; 0.7141 -> 0.6929 | Mixed |
| Subsample models to match Chinchilla model sizes and token counts (ablation) | Gemstones Approach 3 scaling law · Gemstones | slope: 0.6965 -> 0.6315 | Mixed |
| Reduce sampling of Hoffmann data to match the selected models and token counts (ablation) | Hoffmann scaling law, Approach 3 · Hoffmann data | slope: 0.5126 -> 0.6123 | Helped |
| Use the convex hull rather than FLOPs-bin minimizers to fit Approach 1 | Gemstones Approach 1 scaling law · Gemstones | fit reliability | Helped |
| Increase model depth at approximately fixed FLOP budget | Gemstones · Downstream benchmarks | accuracy | Helped |
Surprising findings
- Fitted scaling-law prescriptions can change substantially under subsampling of the same model suite, e.g. a Chinchilla-subsampled fit on the Gemstones data reduced the slope.
- Deeper models give better benchmark accuracy at matched FLOPs, while wider models are faster in wall-clock GPU hours under the tensor-parallel setup used.
Limitations
- Only tensor parallelism is used, so wall-clock conclusions favoring wide models may not generalize to pipeline or other parallelism strategies.
- Scaling laws are expected to decline in applicability for training setups very different from the authors' own.
- Benchmark-accuracy scaling fits are noisy and extrapolate poorly to 2B models, attributed to width/depth variation.
Experimental setup
Datasets: Dolma 1.7 DCLM-baseline FineWeb FineWeb-Edu
Benchmarks: MMLU WinoGrande OpenBookQA ARC CommonSenseQA PIQA SIQA HellaSwag
Metrics: validation log perplexity top-1 error accuracy absolute relative error FLOPs overspend GPU hours
Baselines: Chinchilla (Hoffmann et al.) Kaplan et al. scaling laws Approach 1 binning (Hoffmann et al.) Approach 3 parametric fit
Builds on: Chinchilla Kaplan scaling laws Gemma OLMo Pythia
Closest papers by meaning
- Scaling Laws for Sparsely-Connected Foundation Models (ICLR 2024 · similarity 0.86)
- TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters (ICLR 2025 · similarity 0.84)
- Power Lines: Scaling laws for weight decay and batch size in LLM pre-training (NeurIPS 2025 · similarity 0.84)
- Understanding Scaling Laws with Statistical and Approximation Theory for Transformer Neural Networks on Intrinsically Low-dimensional Data (NeurIPS 2024 · similarity 0.84)
- Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilities (NeurIPS 2025 · similarity 0.83)
- Learning Rate Transfer for Hybrid Transformer-SSM Architectures (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.