Research map / Architectures and efficiency / Attention and transformers
RingAttention with Blockwise Transformers for Near-Infinite Context
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Ring Attention, which distributes long sequences across devices in a ring and overlaps key-value block communication with blockwise attention/FFN computation, enabling context lengths scaling with device count without approximation.
Problem
Transformers' memory cost grows with sequence length, and even memory-efficient blockwise transformers must store each layer's output, limiting context length to what fits on a single device.
Approach
Distribute the outer loop of blockwise attention and FFN across a ring of hosts, rotating key-value blocks between neighbors while overlapping their transfer with blockwise computation, so per-device memory is independent of total sequence length and communication adds no overhead when block compute time exceeds transfer time.
Key results
- Ring Attention supports 4096K context with 32x A100 InfiniBand for 7B, a 32x improvement over the previous best, and 8192K for TPUv4-1024 7B versus 16K-scale baselines (~512x).
- On ExoRL six tasks, AT + Ring Attention achieves a total average return of 113.66 versus 111.13 for AT + BPT with 32 trajectories; BPT and ME run out of memory at 128 trajectories.
- Finetuned LLaMA-13B to 512K context on 32 A100 GPUs and reports line retrieval accuracy maintained at long contexts.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Ring Attention instead of memory-efficient attention and FFN | LLaMA 3B on 8x A100 NVLink · maximum-context training | max context size (×1e3 tokens): 64 -> 512 | Helped |
| Use Ring Attention instead of memory-efficient attention and FFN | LLaMA 7B on 32x A100 InfiniBand · maximum-context training | max context size (×1e3 tokens): 128 -> 4096 | Helped |
| Use Ring Attention instead of memory-efficient attention and FFN | LLaMA 7B on TPUv3-512 · maximum-context training | max context size (×1e3 tokens): 8 -> 2048 | Helped |
| Use Ring Attention instead of memory-efficient attention and FFN | LLaMA 3B on TPUv4-1024 · maximum-context training | max context size (×1e3 tokens): 32 -> 16384 | Helped |
| Use Ring Attention instead of memory-efficient attention and FFN | LLaMA 3B on TPUv5e-256 · maximum-context training | max context size (×1e3 tokens): 32 -> 4096 | Helped |
| Increase trajectories from 32 to 128 and use Ring Attention instead of BPT | AT on ExoRL · Walker Stand | cumulative return: 95.45 -> 98.23 | Helped |
| Increase trajectories from 32 to 128 and use Ring Attention instead of BPT | AT on ExoRL · Walker Run | cumulative return: 105.88 -> 110.45 | Helped |
| Increase trajectories from 32 to 128 and use Ring Attention instead of BPT | AT on ExoRL · Walker Walk | cumulative return: 78.56 -> 78.95 | Helped |
| Increase trajectories from 32 to 128 and use Ring Attention instead of BPT | AT on ExoRL · Cheetah Run | cumulative return: 178.75 -> 181.34 | Helped |
| Increase trajectories from 32 to 128 and use Ring Attention instead of BPT | AT on ExoRL · Jaco Reach | cumulative return: 87.56 -> 89.51 | Helped |
| Increase trajectories from 32 to 128 and use Ring Attention instead of BPT | AT on ExoRL · Cartpole Swingup | cumulative return: 120.56 -> 123.45 | Helped |
| Increase trajectories from 32 to 128 and use Ring Attention instead of BPT | AT on ExoRL · Total Average | cumulative return: 111.13 -> 113.66 | Helped |
Surprising findings
- Memory-efficient baselines run out of memory at 128 trajectories in the ExoRL experiments, while Ring Attention runs them without approximation.
Limitations
- Context-length gains are measured mainly as maximum supported size rather than downstream quality, and the LLM finetune was limited to 512K tokens due to compute budget.
- Overlap requires block compute to exceed transfer time, so lower-bandwidth interconnects (e.g., InfiniBand) need larger blocks and stricter sequence lengths.
Experimental setup
Datasets: ShareGPT ExoRL
Benchmarks: ExoRL line retrieval
Metrics: maximum context size (tokens) model flops utilization cumulative return retrieval accuracy interconnect bandwidth utilization
Baselines: vanilla Transformer memory efficient attention blockwise parallel transformers BC Decision Transformer Algorithm Distillation GPT-3.5-turbo-16K Claude-2-100K Vicuna-16K
Builds on: blockwise parallel transformers memory efficient attention FSDP Algorithm Distillation LLaMA
Closest papers by meaning
- Transformer Block Coupling and its Correlation with Generalization in LLMs (ICLR 2025 · similarity 0.86)
- Dynamical Properties of Tokens in Self-Attention and Effects of Positional Encoding (NeurIPS 2025 · similarity 0.86)
- The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training (ICML 2025 · similarity 0.86)
- Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling (NeurIPS 2024 · similarity 0.86)
- PaTH Attention: Position Encoding via Accumulating Householder Transformations (NeurIPS 2025 · similarity 0.86)
- StarTrail: Concentric Ring Sequence Parallelism for Efficient Near-Infinite-Context Transformer Model Training (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.