AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Fast Attention Over Long Sequences With Dynamic Sparse Flash Attention

Matteo Pagliardini, Daniele Paliotta, Martin Jaggi, François Fleuret

NeurIPS 2023 · Architectures and efficiency › Attention and transformers · approach: Linear attention approximations · 1 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace Reformer chunked LSH attention with exact Hash-sparse kernelGPT2 on MNIST · MNISTperplexity: 1.76 -> 1.67Improved
replace Reformer chunked LSH attention with exact Hash-sparse kernelGPT2 on enwik8 · enwik8 T=4096bits/char: 3.32 -> 2.29Improved
replace FlashAttention with Hash-sparse attention (16 buckets) at T=8192OpenWebText2 LM · OpenWebText2 T=8192iteration speed: 1.8×Improved
increase sequence length from 8192 to 16384 for Hash-sparse (ablation)H-LM · OpenWebText2iteration speedup: 1.8× -> 2.3×Improved
use Hash-sparse attention instead of FlashAttentionOpenWebText2 LM · OpenWebText2validation perplexity per iterationNo clear effect
drop 30% of keys/queries heads randomly (ablation)D-LM · OpenWebText2 T=8192perplexity per iterationNo clear effect
drop 30% of keys/queries heads randomly (ablation)D-LM · OpenWebText2 T=8192training speed gain: ×1.9Improved
increase dropping from 30% to 50% to 70% (ablation)D-LM · OpenWebText2 T=8192training speed gain: ×1.9 -> ×2.6 -> ×3.5Improved
increase dropping to high sparsity (ablation)D-LM · OpenWebText2 T=8192perplexity per iterationHurt
more sparsity in dropping (ablation)D-LM · OpenWebText2 T=8192time to reach perplexityNo clear effect
naive PyTorch implementation of QK dropping instead of SCFA kernel (ablation)QK-sparse attention · runtime benchmarkruntime vs FlashAttention: only faster when dropping more than 70%Hurt
Reformer fixed-chunk attention vs exact hash-collision coverage (ablation)Reformer attention · hash coveragecoverage of collisionsHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy