Research map / Architectures and efficiency / Attention and transformers
Sequence Accumulation and Beyond: Infinite Context Length on Single GPU and Large Clusters
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Sequence Accumulation (SA) and its pipeline-parallel extension SAPP, which split long sequences into sub-sequences processed sequentially while accumulating linear-attention memory states, giving constant memory for training linear sequence models at effectively unbounded context length.
Problem
Linear sequence models have linear memory in context length during training, so very long sequences are hard to train on limited GPUs; existing sequence/context parallelism does not fix this and adds communication overhead.
Approach
Exploit the recurrent form of linear sequence modeling to split a sequence into fixed-length sub-sequences processed one after another on the same device, accumulating a d×d memory state (with backward pass over stored states), and combine this with pipeline parallelism so state accumulation stays on each device with no extra sequence-dimension communication.
Key results
- SA on a single GPU keeps per-GPU memory constant (about 33-35 GB for a 1B model) while context grows from 2K to 1024K.
- Standard PP runs out of memory at 32K context on 4 GPUs, whereas SAPP keeps memory constant across contexts up to 1024K.
- SA, SAPP and their DP/PP combinations reach comparable training loss (3.705-3.714) versus the baseline's 3.710, and downstream benchmark scores are close to baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Sequence Accumulation instead of the baseline training setup | 0.4B basic linear transformer · 16K sequence | loss: 3.710 -> 3.705 | No clear effect |
| Use Sequence Accumulation with Pipeline Parallelism instead of the baseline training setup | 0.4B basic linear transformer · 16K sequence | loss: 3.710 -> 3.712 | No clear effect |
| Use Sequence Accumulation instead of the baseline training setup | 0.4B basic linear transformer · LMB. | score: 30.9 -> 30.7 | Hurt |
| Use Sequence Accumulation with Pipeline Parallelism instead of the baseline training setup | 0.4B basic linear transformer · LMB. | score: 30.9 -> 30.6 | Hurt |
| Use Sequence Accumulation instead of the baseline training setup | 0.4B basic linear transformer · PIQA | score: 64.8 -> 65.3 | Helped |
| Use Sequence Accumulation with Pipeline Parallelism instead of the baseline training setup | 0.4B basic linear transformer · PIQA | score: 64.8 -> 65.6 | Helped |
| Use Sequence Accumulation instead of the baseline training setup | 0.4B basic linear transformer · Hella. | score: 36.1 -> 36.5 | Helped |
| Use Sequence Accumulation with Pipeline Parallelism instead of the baseline training setup | 0.4B basic linear transformer · Hella. | score: 36.1 -> 35.8 | Hurt |
| Use Sequence Accumulation instead of the baseline training setup | 0.4B basic linear transformer · Wino. | score: 50.5 -> 49.6 | Hurt |
| Use Sequence Accumulation with Pipeline Parallelism instead of the baseline training setup | 0.4B basic linear transformer · Wino. | score: 50.5 -> 49.9 | Hurt |
| Use Sequence Accumulation instead of the baseline training setup | 0.4B basic linear transformer · ARC-e | score: 42.9 -> 42.2 | Hurt |
| Use Sequence Accumulation with Pipeline Parallelism instead of the baseline training setup | 0.4B basic linear transformer · ARC-e | score: 42.9 -> 42.7 | Hurt |
Limitations
- Iteration time grows rapidly with context length because sub-sequences are processed sequentially, and throughput only modestly increases.
- Evaluated mainly on small linear transformers (0.4B, 1B, 7B) with pretraining loss and small benchmark gains; no comparison to other long-context methods on downstream long-context tasks.
- Only GPipe is analyzed as the pipeline-parallel scheme; more complex PP methods are left out of scope.
Experimental setup
Datasets: The Pile
Benchmarks: LAMBADA PIQA HellaSwag WinoGrande ARC-e ARC-c
Metrics: training loss GPU memory (GB) throughput (tokens/s) iteration time (ms) accuracy
Baselines: PP DP Baseline (no SA/PP/DP)
Builds on: linear attention GPipe Megatron-LM LASP gradient accumulation
Closest papers by meaning
- ZeCO: Zero-Communication Overhead Sequence Parallelism for Linear Attention (NeurIPS 2025 · similarity 0.88)
- MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (NeurIPS 2024 · similarity 0.87)
- Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequences (ICML 2024 · similarity 0.87)
- FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning (ICLR 2024 · similarity 0.86)
- A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.86)
- Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernels (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.