Research map / Large language models / Language, safety and interpretability
Beyond URLs: Metadata Diversity and Position for Efficient LLM Pretraining
Dongyang Fan, Diba Hashemi, Sai Praneeth Karimireddy, Martin Jaggi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| prepend URL metadata | 1.5B Llama · 9-task downstream average | average accuracy (%): 46.7 -> 47.7 | Improved |
| prepend fine-grained quality score | 1.5B Llama · 9-task downstream average | average accuracy (%): 46.7 -> 47.3 | Improved |
| prepend fine-grained domain information | 1.5B Llama · 9-task downstream average | average accuracy (%): 46.7 -> 47.3 | Improved |
| prepend coarse-grained quality score | 1.5B Llama · 9-task downstream average | average accuracy (%): 46.7 -> 46.6 | No clear effect |
| prepend coarse-grained domain information | 1.5B Llama · 9-task downstream average | average accuracy (%): 46.7 -> 46.7 | No clear effect |
| prepend five learnable meta tokens | 1.5B Llama · 9-task downstream average | average accuracy (%): 46.7 -> 47.1 | Improved |
| append fine-grained domain information | 1.5B Llama · 9-task downstream average | average accuracy (%): 46.7 -> 47.3 | Improved |
| append coarse-grained quality score | 1.5B Llama · 9-task downstream average | average accuracy (%): 46.7 -> 47.1 | Improved |
| append URL | 1.5B Llama · 9-task downstream average | average accuracy (%): 46.7 -> 46.8 | No clear effect |
| append fine-grained quality score | 1.5B Llama · 9-task downstream average | average accuracy (%): 46.7 -> 46.8 | No clear effect |
| prepend full URL instead of only URL prefix (ablation) | URL prepending · 9-task downstream average | average accuracy (%): 46.6 -> 47.7 | Improved |
| prepend full URL instead of only URL domain (ablation) | URL prepending · 9-task downstream average | average accuracy (%): 47.2 -> 47.7 | Improved |
Closest papers by meaning
- URLs Help, Topics Guide: Understanding Metadata Utility in LLM Training (NeurIPS 2025 · similarity 0.94)
- Metadata Conditioning Accelerates Language Model Pre-training (ICML 2025 · similarity 0.88)
- Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.88)
- Improving Pretraining Data Using Perplexity Correlations (ICLR 2025 · similarity 0.87)
- Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum (NeurIPS 2024 · similarity 0.87)
- Enhancing Multilingual LLM Pretraining with Model-Based Data Selection (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
