Research map / Large language models / Language, safety and interpretability
Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains 18 Parc checkpoints (Transformer, Mamba, RWKV on OpenWebText) and analyzes 1,400+ checkpoints across Pythia, Open-GPT2 and Parc models to show that word-level LM log-probabilities follow consistent phases of overfitting to increasing-order n-grams, with unigram frequency and contextual semantic similarity also explaining behavior.
Problem
How LM word-level behavior, as explained by simple heuristics (unigram frequency, n-gram probability, semantic similarity to context), changes over pretraining and whether this trajectory depends on architecture, training data, or scale.
Approach
Treat LM word log-probabilities as a regression target over unigram, 5-gram and contextual semantic similarity predictors at every checkpoint, tracking coefficient trajectories to reveal consistent behavioral phases across architectures, datasets and scales, and releasing the Parc models and NaWoCo dataset.
Key results
- Three heuristics (unigram, 5-gram, semantic similarity) explain up to 98% of variance in LM word log-probability (R^2 = 0.86-0.98 at peak fit).
- Parc-Pythia, Parc-Mamba and Parc-RWKV show Pearson r >= 0.93 with each other at steps >= 80, indicating consistent trajectories across architectures.
- Across models, unigram coefficient rises then falls while 5-gram coefficient rises in later phases; fit stays above R^2 = 0.5 for all models after early training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use unigram probability, 5-gram probability, and semantic similarity together to predict language-model log-probability | Language models tested · NaWoCo | variance explained (R²): R^2=0.86-0.98 | Helped |
| Use unigram probability, 5-gram probability, and semantic similarity to predict log-probability after early training | Language models tested · NaWoCo | variance explained (R²): does not fall below R^2=0.5 | Helped |
| Train through pretraining steps | Autoregressive language models across architectures, scales, and datasets · NaWoCo | correlation with higher-order n-gram probability | Helped |
| Train larger rather than smaller models | Pythia models · NaWoCo | correlation with lower-order n-gram probability | Hurt |
| Train larger rather than smaller models | Pythia models · NaWoCo | 5-gram regression coefficient | Helped |
| Use Common-Crawl-based rather than Wikipedia-based fastText vectors | Contextual semantic similarity predictor · NaWoCo | correlation with language-model log-probability | Helped |
| Use SGPT weighting rather than uniform weighting for Wikipedia-based fastText context vectors (ablation) | Contextual semantic similarity predictor · NaWoCo | correlation with language-model log-probability | Helped |
| Use SGPT weighting rather than uniform weighting for context vectors (ablation) | Contextual semantic similarity predictor · NaWoCo | regression coefficient patterns | No clear effect |
| Train models with different architectures on the same sequences and steps | Parc-Pythia, Parc-Mamba, and Parc-RWKV · NaWoCo, steps ≥80 | Pearson correlation between model log-probabilities: r\geq0.93 | Helped |
| Use n-grams from a matched rather than unmatched training corpus (ablation) | Three-heuristic regressions · NaWoCo | variance explained (R²) | No clear effect |
| Use Common-Crawl-based rather than Wikipedia-based fastText vectors (ablation) | Three-heuristic regressions · NaWoCo | variance explained (R²) | No clear effect |
Surprising findings
- Smaller models retain a larger unigram coefficient late in training, while larger models shift more toward 5-gram probability.
- Common-Crawl fastText similarity is strongly correlated with unigram probability (r about 0.67-0.69), and its apparent correlation with LM log-probability largely follows frequency.
Limitations
- Mamba and RWKV analyzed only at small scale (130-169M parameters) trained on about 2B tokens.
- Only n-grams up to 5 and static fastText embeddings are considered; contextual embeddings not tested.
- Regressions do not account for all variance, especially in the earliest training steps.
Experimental setup
Datasets: OpenWebText The Pile FineWeb
Benchmarks: NaWoCo Paloma LAMBADA SCiQ ARC-Easy SWAG BLiMP
Metrics: Pearson correlation Spearman correlation regression coefficients R^2
Baselines: Pythia Open-GPT2
Builds on: Pythia Mamba RWKV infini-gram SGPT
Closest papers by meaning
- PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs (ICLR 2025 · similarity 0.89)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.89)
- Observational Scaling Laws and the Predictability of Langauge Model Performance (NeurIPS 2024 · similarity 0.89)
- Arrows of Time for Large Language Models (ICML 2024 · similarity 0.89)
- Language Models Can Predict Their Own Behavior (NeurIPS 2025 · similarity 0.89)
- Spurious Forgetting in Continual Learning of Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.