Research map / Large language models / Language, safety and interpretability
Facts in Stats: Impacts of Pretraining Diversity on Language Model Generalization
Tina Behnia, Puneesh Deora, Christos Thrampoulidis
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| raise diversity level DIV (templates per fact) (ablation) | 4-layer Transformer on synthetic facts+stats · ID factual accuracy | iterations to perfect recall | Hurt |
| use very low diversity (DIV<=0.2) (ablation) | 4-layer Transformer · OOD factual recall, MC1POS10 and MC10POS10 | factual accuracy | Hurt |
| use very low diversity with statistical variation across templates (DIV<0.3) (ablation) | 4-layer Transformer · OOD statistical loss | Lossstat | Hurt |
| fix fact positions across templates (MC10POS1) under low diversity (ablation) | 4-layer Transformer · OOD factual recall | factual accuracy | Improved |
| train longer with high diversity (ablation) | 4-layer Transformer · OOD factual accuracy at 3K/4K/16K iterations | factual accuracy: intermediate DIV best at 3K; high DIV catches up by 16K | Mixed |
| retrain only unembedding on high-diversity data using low-diversity features (ablation) | MC10POS10 DIV=0.1 · probing | all metrics | No clear effect |
| freeze features from high-diversity model and retrain unembedding on low-diversity data (ablation) | MC10POS10 · probing | all metrics | Improved |
| transfer attention patterns or token embeddings from high-diversity model (ablation) | MC10POS10 DIV=0.1 · statistical and positional generalization | Accpos, Lossstat | Improved |
| transfer unembedding alone from high-diversity model (ablation) | MC10POS10 DIV=0.1 · factual recall | Accfact | No clear effect |
| transfer unembedding together with attention or embeddings (ablation) | MC10POS10 DIV=0.1 · factual recall | Accfact | Improved |
| use larger learning rate (1e-1) for unembedding layer vs 1e-3 for body (ablation) | 4-layer Transformer, high diversity · factual accuracy convergence | iterations to perfect accuracy | Improved |
| increase MC order, vocabulary size or sequence length (ablation) | 4-layer Transformer, DIV=0.8 · learning speed | Lossstat, Lossfact | Hurt |
Closest papers by meaning
- Generalization v.s. Memorization: Tracing Language Models’ Capabilities Back to Pretraining Data (ICLR 2025 · similarity 0.89)
- Co-occurrence is not Factual Association in Language Models (NeurIPS 2024 · similarity 0.88)
- Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts (ICML 2025 · similarity 0.88)
- Scalable Influence and Fact Tracing for Large Language Model Pretraining (ICLR 2025 · similarity 0.88)
- Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning (NeurIPS 2025 · similarity 0.87)
- SimpleStrat: Diversifying Language Model Generation with Stratification (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
