Research map / Large language models / Instruction and fine-tuning
Skrull: Towards Efficient Long Context Fine-tuning through Dynamic Data Scheduling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Skrull is a dynamic data scheduler for long-context SFT that combines per-micro-batch context-parallel sharding decisions (DACP) with global batch scheduling (GDS) and a lightweight heuristic, implemented on DeepSpeed, yielding 3.76x average speedup over DeepSpeed.
Problem
Long-context SFT trains on mixed long and short sequences, and existing training systems cannot be efficient for both at once: context parallelism and memory-saving settings sized for the longest sequence hurt short sequences, and attention's quadratic compute creates load imbalance across workers.
Approach
Treat parallelism and batching as a per-iteration scheduling problem: shard only sequences that need it (DACP), keep short sequences local within a shared CP group, and globally pack sequences into micro-batches (GDS), solved online by a lightweight heuristic with roll-back to avoid OOM rather than an exact solver.
Key results
- Skrull outperforms DeepSpeed by 3.76x on average and up to 7.54x across real-world long-SFT workloads.
- Average speedups are 5.50x for Qwen2.5-0.5B and 2.03x for Qwen2.5-7B.
- Skrull outperforms the sorted batching baseline by 3.45x on average (peak 6.85x).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Skrull's dynamic data scheduling instead of DeepSpeed | Long-SFT training · Real-world Long-SFT scenarios | speedup: 3.76x on average; up to 7.54x | Helped |
| Use Skrull instead of sorted batching | Long-SFT training · Real-world Long-SFT scenarios | speedup: 3.45x on average; peak improvement of 6.85x | Helped |
| Enable DACP (ablation) | Skrull · Long-SFT training | end-to-end training performance | Helped |
| Enable GDS (ablation) | Skrull · Long-SFT training | end-to-end training performance | Helped |
| Increase BatchSize (ablation) | Skrull on Qwen2.5-0.5B · ChatQA2-Long-SFT | speedup: from 8 to 54 | Helped |
| Increase BucketSize (ablation) | Skrull on Qwen2.5-0.5B · ChatQA2-Long-SFT | speedup: from 8K to 32K | Helped |
| Add the roll-back mechanism to RR scheduling (ablation) | RR on Qwen2.5-0.5B · ChatQA2-Long-SFT | OOM: OOM -> 1.17$\times$ speedup | Helped |
| Add the roll-back mechanism to Skrull (ablation) | Skrull on Qwen2.5-0.5B · ChatQA2-Long-SFT | OOM: OOM -> 1.40$\times$ speedup | Helped |
Limitations
- Speedup on Qwen2.5-7B with ChatQA2-Long-SFT is limited because the major sequence lengths exceed the BucketSize.
- Larger BucketSize improves speed but raises OOM risk, so it requires careful profiling.
- Evaluation covers only throughput and loss equivalence; no downstream task accuracy is reported.
Experimental setup
Datasets: Wikipedia LMsysChat1M ChatQA2-Long-SFT
Benchmarks: Wikipedia LMsysChat1M ChatQA2-Long-SFT
Metrics: average iteration time end-to-end speedup peak memory
Baselines: DeepSpeed (ZeRO-2) sorted batching (LongAlign) round-robin scheduling
Builds on: DeepSpeed FlashAttention LongAlign sorted batching
Closest papers by meaning
- Efficient Long Context Fine-tuning with Chunk Flow (ICML 2025 · similarity 0.91)
- DELIFT: Data Efficient Language model Instruction Fine-Tuning (ICLR 2025 · similarity 0.90)
- Data Mixing Optimization for Supervised Fine-Tuning of Large Language Models (ICML 2025 · similarity 0.90)
- LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models (ICLR 2024 · similarity 0.89)
- SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Models (NeurIPS 2024 · similarity 0.89)
- Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.