Research map / Large language models / Language, safety and interpretability
Fewer Truncations Improve Language Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Best-fit Packing, which splits documents only at the context-length limit and packs the resulting chunks with an optimized best-fit-decreasing bin packing algorithm, so LLM pre-training avoids unnecessary truncation at near-concatenation efficiency.
Problem
Concatenate-then-split data formatting for LLM pre-training fragments many documents into incomplete pieces, causing information loss and closed-domain hallucination.
Approach
Segment documents into chunks at most the context length, then pack chunks into sequences via an optimized Best-Fit-Decreasing bin packing (segment-tree search, O(N log L)), with cross-document attention masked, so only unavoidable truncations occur while padding and training cost stay near concatenation.
Key results
- Best-fit Packing improves average reading comprehension accuracy from 46.71% to 48.92% at 2k length (about +4.7% relative).
- On HumanEval, Pass@100 improves by +15.0% relative and undefined-name hallucination falls by up to 58.3%.
- Packing adds only 0.0024% more training sequences than concatenation at 2k context on RefinedWeb, so training efficiency is nearly preserved.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace concatenation with Best-fit Packing | 13B language model pre-training · Reading comprehension, 2k | Average score: 46.71% -> 48.92% | Helped |
| Replace concatenation with Best-fit Packing | 13B language model pre-training · Reading comprehension, 8k | Average score: 47.13% -> 49.03% | Helped |
| Replace concatenation with Best-fit Packing | 13B language model pre-training · NQ-Swap, 2k | Exact match: 45.62% -> 51.03% | Helped |
| Replace concatenation with Best-fit Packing | 13B language model pre-training · HumanEval, 2k | Pass@100: 35.28% -> 40.57% | Helped |
| Replace concatenation with Best-fit Packing | 7B code language model pre-training · HumanEval, 2k | Undefined name errors: 5.10% -> 2.41% | Helped |
| Replace concatenation with Best-fit Packing | 7B code language model pre-training · MBPP, 2k | Pass@100: 59.46% -> 62.93% | Helped |
| Add cross-document attention mask to concatenation (ablation) | 13B language model pre-training · Validation set | Perplexity: 9.64 -> 9.53 | Helped |
| Add cross-document attention mask to concatenation (ablation) | 13B language model pre-training · Summarization, 2k | ROUGE-2: 12.16 -> 11.79 | Hurt |
| Add cross-document attention mask to concatenation (ablation) | 13B language model pre-training · Reading comprehension, 2k | Average score: 46.71% -> 47.92% | Helped |
| Add cross-document attention mask to concatenation (ablation) | 13B language model pre-training · Context following, 2k | Average score: 40.60% -> 42.41% | Helped |
Surprising findings
- Using a cross-document attention mask on top of concatenation can degrade summarization performance, so packing's gains are not only due to masking.
- Gains on general commonsense/closed-book QA are small (average 64.38% to 64.84%), as common knowledge is redundant across documents.
Limitations
- Pretraining-only experiments; applicability to finetuning is not evaluated.
- Gains on some tasks (e.g., Natural Questions, multiple-choice, commonsense) are small or not significant.
- Hallucination evaluation for summarization relies on imperfect metrics (SummaC, QAFactEval, FAVA).
Experimental setup
Datasets: RefinedWeb The Stack
Benchmarks: NarrativeQA QuAC Natural Questions SQuAD DROP BoolQ RACE MNLI RTE NQ-Swap MemoTrap CNN/DailyMail
Metrics: F1 exact match accuracy ROUGE SummaC QAFactEval FAVA Pass@k undefined name error rate perplexity
Baselines: Concatenation (concat-then-split)
Builds on: LLaMA Best-Fit-Decreasing FlashAttention2 RefinedWeb The Stack
Closest papers by meaning
- Closing the Curious Case of Neural Text Degeneration (ICLR 2024 · similarity 0.88)
- Structured Packing in LLM Training Improves Long Context Utilization (AAAI 2025 · similarity 0.88)
- Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum (NeurIPS 2024 · similarity 0.88)
- Error Norm Truncation: Robust Training in the Presence of Data Noise for Text Generation Models (ICLR 2024 · similarity 0.87)
- Compact Language Models via Pruning and Knowledge Distillation (NeurIPS 2024 · similarity 0.86)
- The Hyperfitting Phenomenon: Sharpening and Stabilizing LLMs for Open-Ended Text Generation (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.