Research map / Large language models / Language, safety and interpretability
PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Releases PolyPythias, 45 new seeded pre-training runs of the Pythia suite (14M–410M, 9 new seeds per size, ~7k checkpoints) and studies how seed-driven initialisation and batch order affect downstream performance, linguistic representations, and training phases.
Problem
Pre-training stability of decoder-only language models with respect to randomness factors (parameter initialisation and data order) is understudied, and the community lacks multi-seed resources to study it.
Approach
Extend the Pythia suite with nine additional seeds per model size (14M to 410M), enabling multi-seed analyses of downstream accuracy, probing-based representational stability, and HMM-based training maps to find outlier runs and predict final performance.
Key results
- Across 10 seeds per size, downstream accuracy is largely consistent; only Pythia-410M seeds 3 and 4 were flagged as outliers, both showing loss spikes.
- Inter-seed agreement peaks at kappa ≈ 0.7 around step 10^3 and settles near 0.5 from step 10^4 onward.
- Bag-of-states from training maps predicts final average z-score of Pythia-410M with R^2 = 0.99 (size-specific HMM), and cross-size HMMs give R^2 > 0.9 for all sizes except 14M.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the Pythia-410M bag-of-states training map to predict its final performance | Pythia-410M · ARC (Easy), LAMBADA, Piqa, and SciQ average accuracy z-score | R²: R²=0.99 | Helped |
| Use a smaller model's HMM to construct the Pythia-410M training map | Pythia-410M performance prediction · ARC (Easy), LAMBADA, Piqa, and SciQ average accuracy z-score | R²: R²>0.9 for all models, except Pythia-14M | Mixed |
| Use only a partial training run to form the bag-of-states (ablation) | Pythia-410M performance prediction · ARC (Easy), LAMBADA, Piqa, and SciQ average accuracy z-score | Prediction accuracy: at least 120 thousand steps (out of the total 143 thousand) | Hurt |
| Enter training-map forks instead of following a linear training map | Pythia-410M seeds 3 and 4 · Downstream performance | Average accuracy | Hurt |
| Increase model size | Pythia models · Downstream multiple-choice tasks | Accuracy | Helped |
| Continue training smaller Pythia models through the later training stage | Pythia models other than Pythia-410M · Downstream performance | Accuracy: between step 10^4 and 10^5 | Hurt |
| Use parameter-statistics training maps from smaller models to predict larger-model training maps | Pythia-410M · Representational stability across model sizes | Pearson correlation: 0.99 for macro-F1, 0.98 for Codelength Ratio, and 0.94 for SSA | Helped |
| Fail to make the state-two-to-state-three transition | Pythia-410M seeds 3 and 4 · Downstream performance | Average accuracy | Hurt |
| Vary the training seed | Pythia models · Downstream performance across seeds | Accuracy | Mixed |
Surprising findings
- Outlier seeds of Pythia-410M diverge from other runs long before worse benchmark scores appear; their training maps contain forks not present in stable runs.
- Training maps from smaller models can zero-shot predict the final performance of the 410M model's seeds.
Limitations
- Model sizes only go up to 410M parameters, so findings may not transfer to larger models.
- Using full training maps to predict performance requires ~120k of 143k steps; early prediction from partial runs failed.
- Small benchmark set limits statistical power for outlier detection; gender bias measures show poor reliability.
Experimental setup
Datasets: The Pile ARC (Easy) ARC (Challenge) LAMBADA Logiqa Piqa SciQ WinoGrande WSC BLiMP CrowS-Pairs Simple Co-occurrence Bias
Benchmarks: ARC (Easy) ARC (Challenge) LAMBADA Logiqa Piqa SciQ WinoGrande WSC BLiMP (Gender Agreement) CrowS-Pairs (Gender) Simple Co-occurrence Bias
Metrics: accuracy Cohen's kappa (inter-seed agreement, self-consistency) macro-F1 codelength ratio subspace angles R^2 of regression z-score
Baselines: MultiBERTs GPT-2 multi-seed runs Llama-2-7B multi-seed suite
Builds on: Pythia GPT-NeoX The Pile training maps MDL probing
Closest papers by meaning
- Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale (NeurIPS 2025 · similarity 0.89)
- An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence (NeurIPS 2026 · similarity 0.89)
- Through the River: Understanding the Benefit of Schedule-Free Methods for Language Model Training (NeurIPS 2025 · similarity 0.89)
- A Probabilistic Perspective on Unlearning and Alignment for Large Language Models (ICLR 2025 · similarity 0.89)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.89)
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.