Research map / Large language models / Language, safety and interpretability
Scaling Instruction-tuned LLMs to Million-token Contexts via Hierarchical Synthetic Data Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Generates long-context instruction data from short-context LLMs via hierarchical chunk summarization and multi-document concatenation, then fine-tunes LLaMA-3.1-8B-Instruct with stepwise RoPE scaling to a 1M-token context.
Problem
Long-context LLMs lack open instruction-tuning data with contexts beyond 100K tokens, and annotating long-context data is scarce and expensive.
Approach
Use short-context LLMs to summarize chunks hierarchically and generate diverse, ordered QA pairs, then concatenate multiple documents (with revisiting follow-up questions) to build arbitrarily long instruction data, trained via stepwise RoPE scaling (180K→350K→650K→1M).
Key results
- The 180K and 350K models scored 59.45 and 59.26 on InfiniteBench vs 51.31 for LLaMA-3.1-8B-Instruct; the 1M model scored 54.80.
- The 1M model reached 62.95% accuracy at 1M tokens on RULER, versus 48.81% for zero-shot RoPE-scaled LLaMA-3.1-8B-Instruct.
- The 1M model kept LongBench average at 46.42 (baseline 48.11) and MMLU at 65.08 (baseline 68.21).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune with hierarchical and diverse synthetic QA data and extend the context to 180K | LLaMA-3.1-8B-Instruct · InfiniteBench (100K tokens), Average | average score: 51.31 -> 59.45 | Helped |
| Fine-tune with hierarchical and diverse synthetic QA data and extend the context to 1M | LLaMA-3.1-8B-Instruct · RULER, 1 000 000 tokens | accuracy: 48.81 % -> 62.95% | Helped |
| Extend context to 1M using stepwise RoPE scaling and synthetic QA tuning | LLaMA-3.1-8B-Instruct · LongBench (10K tokens), All | score: 48.11 -> 46.42 | Hurt |
| Extend context to 1M using stepwise RoPE scaling and synthetic QA tuning | LLaMA-3.1-8B-Instruct · MMLU, mmlu | score: 68.21 ± 0.37 -> 65.08 ± 0.38 | Hurt |
| Fix question counts instead of randomizing them (ablation) | 180K synthetic QA tuning · InfiniteBench, Average | average score: 58.51 -> 59.45 | Helped |
| Combine hierarchical questions with diverse and complex questions (ablation) | 180K synthetic QA tuning · InfiniteBench, Average | average score: 58.14 -> 59.45 | Helped |
| Add global summarization (ablation) | 180K synthetic QA tuning · InfiniteBench, Average | average score: 58.03 -> 58.58 | Helped |
| Use 1M-context training with different synthetic-data generator sizes | LLaMA-3.1-8B-Instruct · InfiniteBench | average score | Mixed |
Surprising findings
- The 650K and 1M models scored lower on InfiniteBench than the 180K and 350K models, which the authors attribute to multi-node training under-training and communication overhead.
- Smaller generator models (Qwen-2.5-7B, LLaMA-3.1-8B) gave gains comparable to Qwen-2-72B on InfiniteBench at 180K and 350K.
Limitations
- 650K and 1M models were under-trained and underperformed shorter-context models on InfiniteBench, attributed to multi-node training issues.
- Only one base model (LLaMA-3.1-8B-Instruct) was extended, and the 650K/1M models were not fully validated against the ablations.
- Single-document ablations were run at 100K and 180K, not at the 1M scale used for the final model.
Experimental setup
Datasets: Together long books dataset (Long-Data-Collections) RULER InfiniteBench LongBench MMLU
Benchmarks: RULER InfiniteBench LongBench MMLU
Metrics: accuracy average score across tasks effective context length
Baselines: LLaMA-3.1-8B-Instruct gradientai/Llama-3-8B-Instruct-Gradient-1048k zero-shot RoPE scaling input truncation
Builds on: RoPE Positional Interpolation NTK-aware scaling RoPE Self-Instruct Evol-Instruct
Closest papers by meaning
- LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions (NeurIPS 2025 · similarity 0.92)
- LongGenBench: Benchmarking Long-Form Generation in Long Context LLMs (ICLR 2025 · similarity 0.92)
- MAmmoTH2: Scaling Instructions from the Web (NeurIPS 2024 · similarity 0.91)
- LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs (ICLR 2025 · similarity 0.91)
- LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (ICML 2024 · similarity 0.90)
- A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.