Research map / Large language models / Instruction and fine-tuning
Hierachical Balance Packing: Towards Efficient Supervised Fine-tuning for Long-Context LLM
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Hierarchical Balance Packing (HBP), a multi-level data packing and training pipeline for long-context SFT that balances attention computation and reduces sequence-parallel communication overhead, reporting up to 2.4x training speedup with competitive performance.
Problem
Hybrid long- and short-context SFT causes workload imbalance; standard data packing ignores imbalanced attention computation across devices and wasted sequence-parallel communication on short samples.
Approach
Builds multiple packing groups with distinct packing lengths, each with its own optimal sequence-parallel degree and gradient-checkpointing setting, assigns samples to groups, balances batches by attention complexity, and trains with curriculum learning and a global-average-token loss normalizer.
Key results
- On DeepSeek-V2 (236B), HBP reduces training from 57.1 to 23.8 GPU days (2.4x) with general AVE 72.0 vs 71.8 for ISF.
- On LLaMA3.1-8B, HBP achieves 1.4x speedup (5.22 to 3.73 GPU days) with AVE 58.2 vs 56.0 for ISF.
- Ablation: hierarchical packing cuts CR from 1.0 to 0.173 and balance batching cuts ABR from 0.288 to 0.002 on LLaMA3.1-8B.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HBP instead of ISF packing | LLaMA3.1-8B · Tulu3 + Longcite | GPU Days: 5.22~(1.0×) -> 3.73~(1.4×) | Helped |
| Use HBP instead of ISF packing | Qwen2.5-32B · Tulu3 + Longcite | GPU Days: 21.3~(1.0×) -> 16.0~(1.33×) | Helped |
| Use HBP instead of ISF packing | LLaMA3.1-70B · Tulu3 + Longcite | GPU Days: 44.4~(1.0×) -> 31.1~(1.42×) | Helped |
| Use HBP instead of ISF packing | DeepSeek-V2 (236B) · Tulu3 + Longcite | GPU Days: 57.1~(1.0x) -> 23.8~(2.4×) | Helped |
| Use HBP instead of ISF packing | LLaMA3.1-8B · General tasks | AVE: 56.0 -> 58.2 | Helped |
| Use HBP instead of ISF packing | LLaMA3.1-8B · LongBench | score: 44.0 -> 43.1 | Hurt |
| Add hierarchical packing (ablation) | LLaMA3.1-8B HBP · Hybrid training | ABR: 0.506 -> 0.288 | Helped |
| Add balance batching to hierarchical packing (ablation) | LLaMA3.1-8B HBP · Hybrid training | ABR: 0.288 -> 0.002 | Helped |
| Add balance batching to hierarchical packing (ablation) | LLaMA3.1-8B HBP · Hybrid training | GPU Days: 4.51~(1.2×) -> 3.73~(1.40×) | Helped |
| Add curriculum learning (ablation) | LLaMA3.1-8B HBP · General tasks | AVE: 56.6 -> 58.2 | Helped |
| Add curriculum learning (ablation) | LLaMA3.1-8B HBP · LongBench | score: 41.6 -> 43.1 | Helped |
| Add curriculum learning (ablation) | LLaMA3.1-8B ISF · LongBench | score: 44.0 -> 43.4 | Hurt |
Surprising findings
- Sum loss normalization causes gradient norms to escalate disproportionately as sequence length grows (up to 1e+5).
- Sample-Mean normalization drops Ruler-128K to 46.1 versus 67.5 for Token-Mean in the loss normalizer ablation.
Limitations
- Experiments not conducted at longer contexts such as 256K or 512K due to compute limits.
- Not validated on other post-training methods such as RLHF or DPO.
- DeepSeek-V2 was trained only at 32K context due to resource constraints.
Experimental setup
Datasets: Tulu3 LongCite OpenHermes LongWriter
Benchmarks: MMLU MMLU-Pro CMMLU BBH MATH GPQA GSM8K HellaSwag MathBench HumanEval MBPP IFEval
Metrics: average accuracy on general tasks long-context scores GPU days ABR CR DBR PR
Baselines: LongAlign ISF packing FlexSP random batching sorted batching
Builds on: LongAlign FlexSP DeepSpeed-Ulysses Megatron-LM ISF packing
Closest papers by meaning
- Structured Packing in LLM Training Improves Long Context Utilization (AAAI 2025 · similarity 0.88)
- Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum (NeurIPS 2024 · similarity 0.88)
- Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining (ICLR 2025 · similarity 0.87)
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.87)
- Efficient Long Context Fine-tuning with Chunk Flow (ICML 2025 · similarity 0.87)
- Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.