Research map / Large language models / Language, safety and interpretability
URLs Help, Topics Guide: Understanding Metadata Utility in LLM Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Systematically studies which metadata (URL, quality score, topic/format domains) helps context-conditioned LLM pretraining and shows context-guided (classifier-free guidance) sampling makes pretrained models more steerable.
Problem
Context-free LLM pretraining discards metadata; prior work on URL conditioning is mixed and lacks understanding of which metadata types help training speed, downstream performance, and controllable generation.
Approach
Pretrain a 1.5B Llama-style model with metadata prepended as masked-loss context (90% context, 10% context-free uniform mixing) and apply classifier-free-guidance-style logit steering at inference to amplify context effects.
Key results
- Only URL conditioning lowers training perplexity from the start; quality score and domain information give no noticeable speedup.
- URL-conditioned model matches the 100B-token standard model's 9-task 5-shot average with only 60B tokens (about 40% speedup); 5-shot avg 47.8 vs 46.7 standard, while 0-shot shows no clear gain.
- Context-guided generation improves LLM-judge scores over context-free sampling for conditioned models (e.g., history URL conditioned 6.73 vs 5.70 context-free), indicating more steerability than standard pretraining.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Prepend URL metadata during pretraining and evaluate with 5-shot prompts | Llama 1.5B context-conditioned pretraining · LM-Eval-Harness 5-shot average | average: 46.7 -> 47.8 | Helped |
| Prepend URL metadata during pretraining and evaluate zero-shot | Llama 1.5B context-conditioned pretraining · LM-Eval-Harness 0-shot average | average: 46.7 -> 46.9 | No clear effect |
| Add quality-score metadata during pretraining | Llama 1.5B context-conditioned pretraining · LM-Eval-Harness 5-shot average | average: 46.7 -> 46.6 | No clear effect |
| Add topic/format domain metadata during pretraining | Llama 1.5B context-conditioned pretraining · LM-Eval-Harness 5-shot average | average: 46.7 -> 46.7 | No clear effect |
| Combine URL and quality-score metadata | Llama 1.5B context-conditioned pretraining · LM-Eval-Harness 5-shot average | average: 46.7 -> 46.1 | Hurt |
| Combine URL and domain-information metadata | Llama 1.5B context-conditioned pretraining · LM-Eval-Harness 5-shot average | average: 46.7 -> 46.7 | No clear effect |
| Combine URL, quality-score, and domain-information metadata | Llama 1.5B context-conditioned pretraining · LM-Eval-Harness 5-shot average | average: 46.7 -> 46.0 | Hurt |
| Add positional <boc><eoc> tokens without metadata (ablation) | Standard pretraining · LM-Eval-Harness 5-shot average | average: 46.7 -> 46.7 | No clear effect |
| Use URL context during inference | URL-conditioned model · Health & medical prompts | LLM judge score: 8.19 ± 0.14 -> 8.14 ± 0.20 | No clear effect |
| Apply classifier-free guidance with URL context | URL-conditioned model · Health & medical prompts | LLM judge score: 8.19 ± 0.14 -> 8.70 ± 0.14 | Helped |
| Use topic/format context during inference | Domain-information-conditioned model · Health & medical prompts | LLM judge score: 8.01 ± 0.16 -> 8.26 ± 0.14 | Helped |
| Apply classifier-free guidance with topic/format context | Domain-information-conditioned model · Health & medical prompts | LLM judge score: 8.01 ± 0.16 -> 9.07 ± 0.05 | Helped |
Surprising findings
- Combining quality score with URL conditioning degrades downstream performance below the standard model.
- The authors could not reproduce the MeCo baseline's reported gains under their evaluation setup.
Limitations
- Only 1.5B models trained on 100B tokens of FineWeb-Edu; generalization to larger scales or other corpora is untested.
- Mechanism behind URL benefit is unclear; only three metadata types examined.
- Context-guided logits break probability calibration, so likelihood-based evaluation is not possible and evaluation relies on LLM-judge generation scores.
Experimental setup
Datasets: FineWeb-Edu WebOrganizer domain labels
Benchmarks: Arc-Easy Arc-Challenge CommonSenseQA MMLU PIQA SIQA HellaSwag LAMBADA Winogrande
Metrics: accuracy training perplexity LLM judge score (coherence, correctness, reasonableness, rel
Baselines: Standard pretraining Positional token ablation MeCo
Builds on: MeCo classifier-free guidance WebOrganizer Megatron-LM
Closest papers by meaning
- Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG (ICLR 2025 · similarity 0.89)
- Training with “Paraphrasing the Original Text” Teaches LLM to Better Retrieve in Long-Context Tasks (AAAI 2025 · similarity 0.88)
- Metadata Conditioning Accelerates Language Model Pre-training (ICML 2025 · similarity 0.88)
- LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs – No Silver Bullet for LC or RAG Routing (ICML 2025 · similarity 0.87)
- DataMan: Data Manager for Pre-training Large Language Models (ICLR 2025 · similarity 0.87)
- Needle Threading: Can LLMs Follow Threads Through Near-Million-Scale Haystacks? (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.