AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

Facts in Stats: Impacts of Pretraining Diversity on Language Model Generalization

Tina Behnia, Puneesh Deora, Christos Thrampoulidis

ICML 2026 · Large language models › Language, safety and interpretability · approach: Pruning and circuits in LLMs · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
raise diversity level DIV (templates per fact) (ablation)4-layer Transformer on synthetic facts+stats · ID factual accuracyiterations to perfect recallHurt
use very low diversity (DIV<=0.2) (ablation)4-layer Transformer · OOD factual recall, MC1POS10 and MC10POS10factual accuracyHurt
use very low diversity with statistical variation across templates (DIV<0.3) (ablation)4-layer Transformer · OOD statistical lossLossstatHurt
fix fact positions across templates (MC10POS1) under low diversity (ablation)4-layer Transformer · OOD factual recallfactual accuracyImproved
train longer with high diversity (ablation)4-layer Transformer · OOD factual accuracy at 3K/4K/16K iterationsfactual accuracy: intermediate DIV best at 3K; high DIV catches up by 16KMixed
retrain only unembedding on high-diversity data using low-diversity features (ablation)MC10POS10 DIV=0.1 · probingall metricsNo clear effect
freeze features from high-diversity model and retrain unembedding on low-diversity data (ablation)MC10POS10 · probingall metricsImproved
transfer attention patterns or token embeddings from high-diversity model (ablation)MC10POS10 DIV=0.1 · statistical and positional generalizationAccpos, LossstatImproved
transfer unembedding alone from high-diversity model (ablation)MC10POS10 DIV=0.1 · factual recallAccfactNo clear effect
transfer unembedding together with attention or embeddings (ablation)MC10POS10 DIV=0.1 · factual recallAccfactImproved
use larger learning rate (1e-1) for unembedding layer vs 1e-3 for body (ablation)4-layer Transformer, high diversity · factual accuracy convergenceiterations to perfect accuracyImproved
increase MC order, vocabulary size or sequence length (ablation)4-layer Transformer, DIV=0.8 · learning speedLossstat, LossfactHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy