Research map / Large language models / Language, safety and interpretability
Structured Packing in LLM Training Improves Long Context Utilization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SPLiCe, which builds long training samples by retrieving mutually relevant documents (BM25, Contriever, repo structure) and briefly fine-tunes OpenLLaMA 3B/7B and CodeLlama 13B to improve long-context utilization.
Problem
Long-context language models often fail to utilize information in long inputs, especially in the middle; the question is how to organize training documents into training samples to improve long-context utilization.
Approach
Build training samples as trees of retrieved related documents (BM25, Contriever-MSMARCO, or repository structure), flattened into long sequences, so that semantic interdependence within a sample is increased, instead of randomly packing documents with BOS/EOS separators.
Key results
- On TREC, SPLiCe improves 13B FoT accuracy at 32K from 89.2 to 92.4 over Example Packing.
- SPLiCe achieves perfect accuracy on Needle In A Haystack and mitigates lost-in-the-middle on key-value retrieval.
- For 13B model, SPLiCe improves GSM8K from 21.9 (EP) to 23.1, and 1.7 points over the starting checkpoint (21.4).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pack related documents with SPLiCE instead of Example Packing | 3B FoT · TREC 32K | accuracy: 73.9 -> 79.3 | Helped |
| Pack related documents with SPLiCE instead of Example Packing | 7B FoT · TREC 16K | accuracy: 74.0 -> 79.0 | Helped |
| Pack related documents with SPLiCE instead of Example Packing | 13B FoT · DBpedia 16K | accuracy: 95.8 -> 96.8 | Helped |
| Pack related documents with SPLiCE instead of Example Packing | 7B CL · DBpedia 16K | accuracy: 96.2 -> 96.4 | Helped |
| Pack related documents with SPLiCE instead of Example Packing | 13B FoT · GSM8K | accuracy: 21.9 -> 23.1 | Helped |
| Pack related documents with SPLiCE instead of Example Packing | 7B FoT · MMLU All | accuracy: 36.2 -> 36.3 | No clear effect |
| Use SPLiCE-packed C code rather than Example Packing | 270M · arXiv | perplexity: 5.550 { $\pm$ .002} -> 5.463 { $\pm$ .002} | Helped |
| Increase the probability of retrieving a random document from 0% to 90% (ablation) | 270M SPLiCE BM25 · arXiv | perplexity: 5.46 -> 5.55 | Hurt |
| Increase the probability of retrieving a random document from 0% to 90% (ablation) | 270M SPLiCE BM25 · Code Python | perplexity: 2.81 -> 2.92 | Hurt |
| Increase SPLiCE-prepared C data proportion from 25% to 75% (ablation) | 270M SPLiCE BM25 · arXiv | perplexity: 5.43 -> 5.61 | Mixed |
Surprising findings
- Training on code data organized with SPLiCe improves perplexity on natural language arXiv data.
- Within-Domain Example Packing gives no clear benefit over standard Example Packing in fine-tuning.
Limitations
- Study limited to document-level granularity.
- Contriever used zero-shot with first 512 tokens, may be suboptimal versus tuned neural retrievers.
- Highly correlated samples could risk training instability, not observed in experiments.
Experimental setup
Datasets: RedPajama StarCoder StackExchange arXiv
Benchmarks: Needle In A Haystack TREC DBpedia Qasper HotPotQA MMLU GSM8K RULER key-value retrieval
Metrics: accuracy perplexity improvement with bootstrap confidence intervals
Baselines: Example Packing Within-Domain Example Packing starting checkpoint
Builds on: Focused Transformer CodeLlama long-context extension BM25 Contriever
Closest papers by meaning
- Training with “Paraphrasing the Original Text” Teaches LLM to Better Retrieve in Long-Context Tasks (AAAI 2025 · similarity 0.90)
- NExtLong: Toward Effective Long-Context Training without Long Documents (ICML 2025 · similarity 0.90)
- Needle Threading: Can LLMs Follow Threads Through Near-Million-Scale Haystacks? (ICLR 2025 · similarity 0.89)
- Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum (NeurIPS 2024 · similarity 0.89)
- Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG (ICLR 2025 · similarity 0.89)
- Data Engineering for Scaling Language Models to 128K Context (ICML 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.