AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Long-Context Attention Benchmark: From Kernel Efficiency to Distributed Context Parallelism

Tao Bu, Qiangang Wang, Bowen Zeng, Hanwen Sun, Yunpeng Huang, Chun Cao, Jingwei Xu

ICLR 2026 · Architectures and efficiency › Attention and transformers · approach: Sparse attention for long-context LLMs

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
increase block size from 64 to 128 for FlashInfer block sparse kernel (ablation)FlashInfer · Block sparse, GQA 64:8, 16k, sparsity 0.5forward TFLOPs: 236.30 -> 449.79Improved
increase block size from 64 to 128 for FlexAttention sparse (ablation)FlexAttention · Block sparse, 16K, MHA 4:4, sparsity 0.5forward TFLOPs: 48.76 -> 345.84Improved
use larger sparsity ratio in FlashInfer with MHA and 128 block at long context (ablation)FlashInfer · Block sparse, MHA 64:64, block 128forward TFLOPs: 384.05 (SR 0.2, 128k) -> OOM (SR 0.5, 128k)Hurt
switch from MHA to GQA in sparse kernelsFA2 Sparse and FlashInfer · Block sparse kernelsTFLOPsNo clear effect
use hardware-specialized kernel (VSA on Hopper) rather than general kernelsblock sparse attention · Block sparse, MHA, block 64TFLOPsImproved
use FA3 instead of FA2 or cuDNN on H100 for dense attentiondense attention kernels · Dense kernels, 8K, GQA 64:8TFLOPsImproved
use FlexAttention sparse masks instead of dedicated kernels at long contextFlexAttention · Block sparse, 32K-128KmemoryHurt
use hybrid Ulysses plus ring design (USP, LoongTrain) instead of pure Ulysses or pure Ring P2Pcontext parallel attention · Context parallel, FULL DOCUMENT, 8 to 96 GPUsforward TFLOPsImproved
add LoongTrain secondary P2P (DoubleRing) over USPUSP · Context parallel, up to 96 GPUsTFLOPsMixed
increase sequence length on single-GPU dense kernelsdense attention kernels · Dense kernels, 1K to 48KTFLOPsImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy