Research map / Architectures and efficiency / Attention and transformers
Long-Context Attention Benchmark: From Kernel Efficiency to Distributed Context Parallelism
Tao Bu, Qiangang Wang, Bowen Zeng, Hanwen Sun, Yunpeng Huang, Chun Cao, Jingwei Xu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase block size from 64 to 128 for FlashInfer block sparse kernel (ablation) | FlashInfer · Block sparse, GQA 64:8, 16k, sparsity 0.5 | forward TFLOPs: 236.30 -> 449.79 | Improved |
| increase block size from 64 to 128 for FlexAttention sparse (ablation) | FlexAttention · Block sparse, 16K, MHA 4:4, sparsity 0.5 | forward TFLOPs: 48.76 -> 345.84 | Improved |
| use larger sparsity ratio in FlashInfer with MHA and 128 block at long context (ablation) | FlashInfer · Block sparse, MHA 64:64, block 128 | forward TFLOPs: 384.05 (SR 0.2, 128k) -> OOM (SR 0.5, 128k) | Hurt |
| switch from MHA to GQA in sparse kernels | FA2 Sparse and FlashInfer · Block sparse kernels | TFLOPs | No clear effect |
| use hardware-specialized kernel (VSA on Hopper) rather than general kernels | block sparse attention · Block sparse, MHA, block 64 | TFLOPs | Improved |
| use FA3 instead of FA2 or cuDNN on H100 for dense attention | dense attention kernels · Dense kernels, 8K, GQA 64:8 | TFLOPs | Improved |
| use FlexAttention sparse masks instead of dedicated kernels at long context | FlexAttention · Block sparse, 32K-128K | memory | Hurt |
| use hybrid Ulysses plus ring design (USP, LoongTrain) instead of pure Ulysses or pure Ring P2P | context parallel attention · Context parallel, FULL DOCUMENT, 8 to 96 GPUs | forward TFLOPs | Improved |
| add LoongTrain secondary P2P (DoubleRing) over USP | USP · Context parallel, up to 96 GPUs | TFLOPs | Mixed |
| increase sequence length on single-GPU dense kernels | dense attention kernels · Dense kernels, 1K to 48K | TFLOPs | Improved |
Closest papers by meaning
- Curse of High Dimensionality Issue in Transformer for Long Context Modeling (ICML 2025 · similarity 0.92)
- Core Context Aware Transformers for Long Context Language Modeling (ICML 2025 · similarity 0.92)
- A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contexts (ICLR 2025 · similarity 0.91)
- A Training-Free Sub-quadratic Cost Transformer Model Serving Framework with Hierarchically Pruned Attention (ICLR 2025 · similarity 0.91)
- Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibration (ICML 2024 · similarity 0.90)
- Rope to Nope and Back Again: A New Hybrid Attention Strategy (NeurIPS 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
