AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Fast Transformers with Clustered Attention

Apoorv Vyas, Angelos Katharopoulos, François Fleuret

NeurIPS 2020 · Architectures and efficiency › Attention and transformers · approach: Sparse efficient transformers for long context

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add improved clustering (top-k exact keys) on top of clustered attention (ablation)clustered attention · WSJ testphone error rate: 7.50 -> 5.61Improved
use clustered attention instead of fullfull attention · WSJ 9 layerstime per epoch (s): 2514 -> 803Improved
use improved clustered attention instead of fullfull attention · WSJ 9 layersconvergence time (h): 87.99 -> 72.14Improved
use clustered attention instead of Reformer (lsh-4)Reformer · WSJ testphone error rate: 8.59 -> 7.50Improved
use clustered attention instead of fullfull attention · Switchboard testword error rate: 15.0 -> 18.5Hurt
use improved clustered attention instead of fullfull attention · Switchboard testword error rate: 15.0 -> 15.5No clear effect
use improved clustered attention instead of fullfull attention · Switchboardconvergence time (h): 228.05 -> 127.44Improved
use clustered attention instead of fullfull attention · Switchboardtime per epoch (h): 3.84 -> 1.91Improved
increase clusters from 100 to 200 at inferenceclustered attention (full-trained model) · WSJ valphone error rate: 44.88 -> 21.76Improved
add top-k exact keys to clustered attention when approximating fullclustered-100 · WSJ val, full-trainedphone error rate: 44.88 -> 9.29Improved
keep only the 32 highest-attention keys per query (oracle-top) instead of improved clustered (ablation)i-clustered · WSJ val, full-trainedphone error rate: 9.29 -> 17.16Hurt
use improved clustered attention instead of clustered with 25 clustersclustered-25 · SQuADF1: 0.006 -> 0.876Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy