Research map / Large language models / Language, safety and interpretability
Metadata Conditioning Accelerates Language Model Pre-training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MeCo, which prepends source URLs as metadata during most of LM pre-training, then cools down on standard text, yielding faster pre-training and metadata-based steering at inference.
Problem
Pre-training corpora are heterogeneous in style, domain, and quality, and standard pre-training treats all documents identically, making it data-inefficient and hard to steer toward specific behaviors.
Approach
Prepend each document's source URL as metadata during the first 90% of pre-training (computing loss only on document tokens), then run a 10% cooldown on standard data without metadata so the model works without metadata, and use real or fabricated URLs at inference to steer outputs.
Key results
- A 1.6B MeCo model trained on 160B DCLM tokens reaches 56.7 average vs 55.7 for standard pre-training, matching the 240B-token baseline (56.7 avg for standard 240B is reported as 56.7 in the table) with 33% less data.
- Conditional inference with fabricated URLs raises the MeCo 1.6B average from 56.7 to 57.2 while the standard model changes little (55.7 to 55.8).
- Conditioning on en.wikipedia.org reduces toxicity scores of generations, with MeCo plus conditional inference giving a several-fold lower toxicity than standard unconditional inference.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add URL metadata during the first 90% of training and standard-data cooldown for the last 10% | 1.6B model trained on DCLM · DCLM, 160B tokens | 10-task average: 55.7 -> 56.7 | Helped |
| Add URL metadata conditioning then cooldown | 1.6B model · DCLM, 160B vs. 240B tokens | 10-task average: MeCo 56.7; standard 240B tokens 56.7 | Helped |
| Add URL metadata conditioning then cooldown | 1.6B model · C4, 160B tokens | 10-task average: 50.5 -> 51.6 | Helped |
| Add URL metadata conditioning then cooldown | 1.6B model · RefinedWeb, 160B tokens | 10-task average: 53.2 -> 54.0 | Helped |
| Add task-specific URL prefixes at inference | MeCo 1.6B model · DCLM, 160B tokens | 10-task average: 56.7 -> 57.2 | Helped |
| Condition inference on www.factmonster.com instead of using no URL | MeCo 1.6B model · Zero-shot CSQA | accuracy: 54.7 -> 60.9 | Helped |
| Add standard-data cooldown after URL-conditioned training (ablation) | MeCo 1.6B model · DCLM, 160B tokens | 10-task average: 50.3 -> 56.7 | Helped |
| Use sequential URL conditioning then standard-data cooldown rather than mixing both throughout training (ablation) | MeCo 1.6B model · DCLM, 160B tokens | 10-task average: 56.4 -> 56.7 | Helped |
| Increase cooldown from 10% to 20% (ablation) | MeCo 1.6B model · DCLM, 160B tokens | 10-task average: 56.7 -> 56.8 | No clear effect |
| Increase cooldown from 10% to 30% (ablation) | MeCo 1.6B model · DCLM, 160B tokens | 10-task average: 56.7 -> 56.2 | Hurt |
| Replace absolute domain names with URL suffixes (ablation) | MeCo 1.6B model · DCLM, 160B tokens | 10-task average: 56.7 -> 56.2 | Hurt |
| Restrict metadata to only the top 0.2% or top 2% of URLs (ablation) | MeCo 1.6B model · DCLM, 160B tokens | 10-task average: 56.7 -> 56.4; 56.7 -> 56.3 | Hurt |
Surprising findings
- Hashed URLs perform on par with semantically meaningful URLs (56.7 vs 56.7), suggesting grouping by source rather than URL semantics drives the gain.
- Validation perplexity does not correlate with downstream performance; the 240B baseline has much lower perplexity yet matches MeCo's average.
Limitations
- Single run per experiment; variance estimated only from cooldown reruns of baseline (std 0.1%).
- Only English corpora studied and no interplay with post-training examined.
- No mechanistic understanding of why metadata conditioning improves downstream performance.
Experimental setup
Datasets: DCLM-Baseline RefinedWeb (DCLM-reproduced) C4 (SlimPajama subset)
Benchmarks: MMLU ARC-Easy ARC-Challenge CommonsenseQA HellaSwag OpenBookQA PIQA Social IQA WinoGrande TruthfulQA
Metrics: average accuracy over 10 tasks validation perplexity toxicity score (Detoxify)
Baselines: Standard pre-training Data selection (fastText classifier) 100% URL training 90% URL + 10% standard mixing Standard pre-training with 240B tokens
Builds on: CTRL Llama OLMES DCLM
Closest papers by meaning
- URLs Help, Topics Guide: Understanding Metadata Utility in LLM Training (NeurIPS 2025 · similarity 0.88)
- Reusing Pre-Training Data at Test Time is a Compute Multiplier (ICLR 2026 · similarity 0.87)
- Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scale (ICML 2025 · similarity 0.87)
- Predictive Data Selection: The Data That Predicts Is the Data That Teaches (ICML 2025 · similarity 0.87)
- DEPT: Decoupled Embeddings for Pre-training Language Models (ICLR 2025 · similarity 0.87)
- Towards Scalable Data Diversification for Language Model Pretraining via Leverage Score Sampling (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.