Research map / Architectures and efficiency / Attention and transformers
SkyLadder: Better and Faster Pretraining via Context Window Scheduling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that shorter pretraining context windows beat longer ones under a fixed token budget, and proposes SkyLadder, a short-to-long context window schedule that improves standard benchmarks and speeds up training by up to 22%.
Problem
Long-context pretraining is assumed to help, but under a fixed token budget it is unclear whether longer context windows improve or hurt pretraining, and how to balance long-context capability against pretraining effectiveness and efficiency.
Approach
Schedule the attention context window during pretraining from a tiny size (8 tokens) linearly up to the target length (e.g., 8K/32K) by applying block-wise local causal masks to packed sequences, which can be combined with intra-document masking and requires no change to the data.
Key results
- On 1B models pretrained on 100B CommonCrawl tokens, SkyLadder improves the standard-benchmark average from 46.3 to 50.0 (+3.7) over random packing.
- SkyLadder reduces training time by 13% at 8K context and 22% at 32K context, with FLOPs reduced by 14.7% and 26.3% respectively.
- Using FineWeb-Pro with a 32K context, SkyLadder raises the standard average from 50.7 to 54.3 and the long-context average from 9.7 to 13.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Schedule the context window short-to-long instead of keeping it constant long | Random-packed 1B pretraining on CC · Standard benchmarks | average accuracy: 46.3 -> 50.0 { (+3.7)} | Helped |
| Schedule the context window short-to-long with Random packing | 1B pretraining on CC · Reading comprehension benchmarks | average accuracy: 25.5 -> 30.2 { (+4.7)} | Helped |
| Schedule the context window short-to-long with Random packing | 1B pretraining on CC · Long benchmarks | average accuracy: 15.3 -> 14.3 | Hurt |
| Schedule the context window short-to-long with IntraDoc masking | 1B pretraining on CC · Standard benchmarks | average accuracy: 47.4 -> 49.3 { (+1.9)} | Helped |
| Schedule the context window short-to-long with IntraDoc masking | 1B pretraining on CC · Reading comprehension benchmarks | average accuracy: 28.7 -> 29.1 { (+0.4)} | No clear effect |
| Schedule the context window short-to-long with IntraDoc masking | 1B pretraining on CC · Long benchmarks | average accuracy: 13.0 -> 13.2 | No clear effect |
| Schedule context short-to-long rather than constant long at 32K | 1B pretraining on FineWeb-Pro · Standard benchmarks | average accuracy: 50.7 -> 54.3 { (+3.6)} | Helped |
| Schedule context short-to-long rather than constant long at 32K | 1B pretraining on FineWeb-Pro · Long benchmarks | average accuracy: 9.7 -> 13.5 { (+3.8)} | Helped |
| Use short-to-long rather than long-to-short scheduling (ablation) | 1B pretraining on FineWeb-Pro · Long benchmarks | average accuracy: 10.7 -> 12.3 { (+1.2)} | Helped |
| Replace linear scheduling with sinusoidal scheduling (ablation) | 1B pretraining at 32K on FineWeb-Pro · Long benchmarks | average accuracy: 13.5 -> 14.2 | Helped |
| Use exponential instead of linear short-to-long scheduling (ablation) | 1B pretraining at 32K on FineWeb-Pro · Long benchmarks | average accuracy: 13.5 -> 11.5 | Hurt |
| Use continual pretraining with an abrupt 4K-to-32K switch instead of a gradual schedule (ablation) | 1B pretraining at 32K on FineWeb-Pro · Standard benchmarks | average accuracy: 50.7 -> 52.9 | Hurt |
Surprising findings
- Models pretrained with shorter context windows consistently outperform long-context counterparts under a fixed token budget, even on long-context evaluation in some settings.
- A long-to-short schedule with the same data mix does not help, indicating the scheduling itself rather than the data length mixture drives gains.
Limitations
- Experiments limited to up to 3B parameters and 32K context, not larger LLMs.
- No theoretical analysis of why scheduling helps.
- Hyperparameters (alpha, w_s) not extensively searched and single runs without repeated seeds.
Experimental setup
Datasets: SlimPajama CommonCrawl FineWeb-Pro Python code corpus
Benchmarks: HellaSwag ARC-Easy ARC-Challenge CommonsenseQA OpenBookQA PIQA Social-QA Winogrande MMLU HotpotQA SQuAD NQ
Metrics: accuracy perplexity exact match pass@k training time FLOPs
Baselines: Random packing IntraDoc BM25 packing Dataset Decomposition continual pretraining schedule
Builds on: TinyLlama Flash Attention 2 Dataset Decomposition RoPE
Closest papers by meaning
- MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (NeurIPS 2024 · similarity 0.88)
- Maximizing Intermediate Checkpoint Value in LLM Pretraining with Bayesian Optimization (ICML 2025 · similarity 0.88)
- LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens (ICML 2024 · similarity 0.87)
- ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities (ICLR 2025 · similarity 0.87)
- DON’T STOP ME NOW: EMBEDDING BASED SCHEDULING FOR LLMS (ICLR 2025 · similarity 0.87)
- Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.