Research map / Large language models / Language, safety and interpretability
The Coverage Principle: How Pre-Training Enables Post-Training
Fan Chen, Audrey Huang, Noah Golowich, Sadhika Malladi, Adam Block, Jordan T. Ash, Akshay Krishnamurthy, Dylan J Foster
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| continue minimizing cross-entropy in later training on a two-class mixed graph task | GPT-2-style transformer · graph search (layered DAG, mixture of two classes) | coverage profile / Best-of-N | Hurt |
| use a mid-training checkpoint selected by coverage rather than the final low-cross-entropy checkpoint | pre-trained transformer · graph search | coverage profile | Improved |
| increase sequence horizon H (8, 16, 24) | next-token prediction · graph search | KL divergence vs coverage profile | Mixed |
Closest papers by meaning
- An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence (NeurIPS 2026 · similarity 0.88)
- Predictive Data Selection: The Data That Predicts Is the Data That Teaches (ICML 2025 · similarity 0.88)
- Language models scale reliably with over-training and on downstream tasks (ICLR 2025 · similarity 0.88)
- PolyPythias: Stability and Outliers across Fifty Language Model Pre-Training Runs (ICLR 2025 · similarity 0.87)
- Reusing Pre-Training Data at Test Time is a Compute Multiplier (ICLR 2026 · similarity 0.87)
- A Hitchhiker's Guide to Scaling Law Estimation (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
