AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

RACE Attention: A Strictly Linear-Time Attention for Long-Sequence Training

Sahil Joshi, Agniva Chowdhury, Amar Kanakamedala, Ekam Singh, Evan Tu, Anshumali Shrivastava

ICLR 2026 · Architectures and efficiency › Attention and transformers · approach: Linear attention approximations

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
scale sketch size from (P=2,L=2) to (P=4,L=4) (ablation)RACE Attention · WikiText-103 @1024perplexity: 23.9 -> 20.9Improved
increase hash tables L from 3 to 4 at P=3 (ablation)RACE Attention · WikiText-103 @1024perplexity: 21.9 -> 21.5Improved
increase P from 3 to 5 at L=4 (ablation)RACE Attention · Tiny Stories @512perplexity: 3.9 -> 2.7Improved
increase L from 4 to 5 at P=5 (ablation)RACE Attention · Tiny Stories @512perplexity: 2.7 -> 5.1Hurt
increase P and L from (3,3) to (4,4) (ablation)RACE Attention · SST-2 @1024accuracy: 77.1% -> 79.4%Improved
increase P and L from (2,2) to (4,4) (ablation)RACE Attention · CIFAR-10 @1024accuracy: 63.7 -> 62.5 -> 65.9Mixed
sharpen angular kernel from gamma=8 to gamma=12 (ablation)Angular Attention · Penn Tree Bank @128test perplexity: 58.8 -> 57.6Improved
use RACE sketch instead of exact angular attention (gamma=8)Angular Attention · Penn Tree Bank @128test perplexity: 58.8 -> 53.4Improved
use RACE sketch instead of exact angular attention (gamma=8)Angular Attention · Text Retrieval @8000accuracy: OOM -> 80.9%Improved
use RACE instead of FlashAttention2 on 64K ArxivFlashAttention2 · Arxiv long-document classification @64Ktrain time per epoch: 1645s -> 561sImproved
use RACE instead of FlashAttention2 on Food-101FlashAttention2 · Food-101 @16Ktrain time per epoch: 2600s -> 891sImproved
use RACE instead of FlashAttention2 at ~4M tokens on GH200FlashAttention2 · single attention layer fwd-bwd, GH200wall-clock time: 550 seconds -> 0.1 secondsImproved

Earlier methods this paper compares with

Named baselines from earlier accepted papers in this paper's results tables: better than 1 of 1 on most shared results.

Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy