Research map / Large language models / Language, safety and interpretability
Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Data
Haoran Deng, Yingyu Lin, Zhenghao Lin, Xiao Liu, Yizhou Sun, Yian Ma, Yeyun Gong
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| select top 20% of long data by LongFilter score instead of unfiltered long data | Llama-3-8B continued pretraining · LongBench overall (Book) | overall score: 37.47 -> 39.81 | Improved |
| select top 20% by LongFilter score instead of unfiltered data | Llama-3-8B continued pretraining · RULER overall (Book) | overall score: 73.35 -> 78.95 | Improved |
| LongFilter selection vs unfiltered data | Llama-3-8B continued pretraining · LongBench overall (ArXiv) | overall score: 38.58 -> 39.52 | Improved |
| LongFilter selection vs unfiltered data | Llama-3-8B continued pretraining · RULER overall (ArXiv) | overall score: 69.28 -> 70.13 | Improved |
| LongFilter selection vs unfiltered data | Llama-3-8B continued pretraining · RULER overall (CommonCrawl) | overall score: 72.59 -> 75.37 | Improved |
| LongFilter selection vs unfiltered data | Llama-3-8B continued pretraining · RULER Other subtask (CommonCrawl) | score: 47.54 -> 31.71 | Hurt |
| LongFilter selection vs LongWanjuan | Llama-3-8B continued pretraining · LongBench overall (CommonCrawl) | overall score: 40.02 -> 40.66 | Improved |
| LongFilter selection vs unfiltered data | Llama-3-8B continued pretraining · HELMET overall | training efficiency | Improved |
| LongFilter selection vs unfiltered data | Llama-3-8B continued pretraining · HELMET Recall (SubEM) | recall: up to 10% gain at 1B tokens | Improved |
| LongFilter selection vs LongWanjuan | Llama-3-8B continued pretraining · LongBench Summ (ArXiv) | score: 24.83 vs 24.48 | Mixed |
Closest papers by meaning
- NExtLong: Toward Effective Long-Context Training without Long Documents (ICML 2025 · similarity 0.93)
- Structured Packing in LLM Training Improves Long Context Utilization (AAAI 2025 · similarity 0.91)
- Training with “Paraphrasing the Original Text” Teaches LLM to Better Retrieve in Long-Context Tasks (AAAI 2025 · similarity 0.91)
- What is Wrong with Perplexity for Long-context Language Modeling? (ICLR 2025 · similarity 0.90)
- A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.90)
- Data Engineering for Scaling Language Models to 128K Context (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
