Research map / Large language models / Language, safety and interpretability
NExtLong: Toward Effective Long-Context Training without Long Documents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Synthesizes long-context training data by splitting short documents into meta-chunks and interleaving retrieved hard-negative distractor chunks from a FAISS index of the pretraining corpus, enabling strong 128K/512K long-context models without long documents.
Problem
Long-context LLM training normally needs scarce non-synthetic long documents, and existing short-document concatenation synthesis methods (random, KNN, ICLM, Quest) lack an explicit mechanism for learning long-range dependencies.
Approach
Extend each meta-document by interleaving top-k semantically similar but distracting chunks (hard negatives) retrieved from a deduplicated pretraining corpus after each meta-chunk, placing the meta-chunk first, so dependent segments become long-range and the model must discriminate relevant context from distractors under standard next-token prediction.
Key results
- On the 128K setting with Meta-Llama-3-8B, NExtLong averages 62.58 on HELMET+RULER versus 55.25 for Quest and 52.85 for Standard concatenation.
- Llama-3-8B-NExtLong-512K-Base averages 65.76, outperforming Llama-3-8B-ProLong-512K-Base (60.34) and Llama-3.1-8B (61.07) without using non-synthetic long documents.
- Llama-3-8B-NExtLong-512K-Base achieves 100% on Needle-in-a-Haystack and LongBench v2 overall 30.4 vs 27.2 for ProLong-512K-Instruct.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on NExtLong negative-extended documents instead of Quest-synthesized documents | Meta-Llama-3-8B-base · HELMET (8K–128K average) | Avg.: 55.25 -> 62.58 | Helped |
| Train on NExtLong negative-extended documents instead of Quest-synthesized documents | Meta-Llama-3-8B-base · HELMET (8K–128K average) | Recall: 69.13 -> 82.56 | Helped |
| Train on NExtLong negative-extended documents instead of Quest-synthesized documents | Meta-Llama-3-8B-base · HELMET (8K–128K average) | Re-rank: 22.35 -> 31.47 | Helped |
| Train on NExtLong negative-extended documents instead of Standard concatenation | Meta-Llama-3-8B-base · RULER (8K–128K average) | score: 76.68 -> 81.50 | Helped |
| Use NExtLong synthetic training data instead of ProLong long-document training | Llama-3-8B base model · HELMET (8K–128K average) | Avg.: 60.34 -> 65.76 | Helped |
| Use NExtLong rather than ProLong for long-context training | Llama-3-8B Instruct · LongBench v2 | Overall: 27.2 -> 30.4 | Helped |
| Place the meta-chunk before hard negatives rather than after them (ablation) | NExtLong · HELMET (8K–128K average) | Avg.: 60.01 -> 62.58 | Helped |
| Place the meta-chunk before hard negatives rather than inserting it randomly (ablation) | NExtLong · HELMET (8K–128K average) | Avg.: 58.95 -> 62.58 | Helped |
| Combine Cosmopedia and FineWebEdu rather than use Cosmopedia alone (ablation) | NExtLong · HELMET (8K–128K average) | Avg.: 59.26 -> 62.58 | Helped |
| Combine Cosmopedia and FineWebEdu rather than use FineWebEdu alone (ablation) | NExtLong · HELMET (8K–128K average) | Avg.: 62.04 -> 62.58 | Helped |
| Use top-k similar hard negatives instead of low-similarity negatives (ablation) | NExtLong · HELMET and RULER | average performance | Helped |
| Use retrieved hard negatives instead of repeating meta-chunks (ablation) | NExtLong · HELMET and RULER | average performance | Helped |
Surprising findings
- Using repeated meta-chunks as negatives or low-similarity (tail) negatives degrades performance, indicating that false negatives and too-easy distractors both hurt.
- Placing the meta-chunk before its hard negatives (Head) outperforms placing it at the tail or randomly inserted.
Limitations
- Ablations were run only at 128K context due to high computational cost.
- Negative mining relies on a fixed retrieval-based strategy; generative harder distractors were not explored.
- Gains on some HELMET categories (e.g., Re-rank) vs. GPT-4o and Gemini remain well below closed-source models.
Experimental setup
Datasets: Cosmopedia v2 FineWeb-Edu UltraChat
Benchmarks: HELMET RULER LongBench v2 Needle-in-a-Haystack short-text suite (HellaSwag, LAMBADA, ARC, PIQA, WinoGrande, LogiQA)
Metrics: average accuracy/score across HELMET categories RULER score LongBench v2 accuracy attention-based long-dependency probe accuracy
Baselines: Standard concatenation KNN ICLM Quest Yarn-Llama-2-7b-128K DeepSeek-V2-Lite Yi-9B-200K Qwen2.5-7B Mistral-Nemo-Base Llama-3-8B-ProLong-512K-Base Llama-3.1-8B GPT-4o
Builds on: Quest ProLong HELMET RULER
Closest papers by meaning
- LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (ICML 2024 · similarity 0.90)
- Training with “Paraphrasing the Original Text” Teaches LLM to Better Retrieve in Long-Context Tasks (AAAI 2025 · similarity 0.90)
- Structured Packing in LLM Training Improves Long Context Utilization (AAAI 2025 · similarity 0.90)
- A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.90)
- Make Your LLM Fully Utilize the Context (NeurIPS 2024 · similarity 0.89)
- A Controllable Examination for Long-Context Language Models (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.