Research map / Architectures and efficiency / Attention and transformers
SMYRF - Efficient Attention using Asymmetric Clustering
Giannis Daras, Nikita Kitaev, Augustus Odena, Alexandros G Dimakis
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace dense attention with SMYRF at 50% memory | BERT base · GLUE dev | Avg.: $82.69$ -> $83.12$ | Improved |
| Reduce SMYRF-BERT memory to 25% and use cluster size 16 (ablation) | BERT base · GLUE dev | Avg.: $83.12$ -> $81.74$ | Hurt |
| Replace full-length input with 32-token truncation | BERT base · GLUE dev | Avg.: $82.69$ -> $73.56$ | Hurt |
| Replace dense attention with SMYRF at 50% memory | BERT base · IMDB | accuracy: $94.12\%$ -> $92.64\%$ | Hurt |
| Reduce SMYRF-BERT memory from 50% to 25% (ablation) | BERT base · IMDB | accuracy: $92.64\%$ -> $92.52\%$ | Hurt |
| Reduce SMYRF-BERT memory from 25% to 12.5% (ablation) | BERT base · IMDB | accuracy: $92.52\%$ -> $91.46$ | Hurt |
| Train with SMYRF and switch to dense attention for inference | RoBERTa base · IMDB | accuracy: $93.72\%$ -> $94.62\%$ | Improved |
| Train with SMYRF and switch to dense attention for inference | BERT base · IMDB | accuracy: $92.64\%$ -> $93.54\%$ | Improved |
| Replace SMYRF asymmetric hashing with E2LSH (ablation) | SMYRF-RoBERTa · IMDB (3 epochs) | accuracy: 93.7\% -> 89.3\% | Hurt |
| Replace SMYRF hashing with cross-polytope LSH (ablation) | SMYRF-RoBERTa · IMDB (3 epochs) | accuracy: 93.7\% -> 88.7\% | Hurt |
| Use 512-token context rather than Longformer's 4096-token context | SMYRF · Hyperpartisan News Detection | accuracy: $94.8\%$ -> $97.2\%$ | Improved |
| Replace dense attention with SMYRF at 50% memory | BigGAN · BigGAN, ImageNet | Inception: $93.79 \pm 1.96$ -> $92.09 \pm 1.83$ | Hurt |
Later papers that compared with this method
1 later accepted paper reports SMYRF as a baseline in their results tables; 1 of them report a better number on most of the results they share with it.
- Scatterbrain: Unifying Sparse and Low-rank Attention (NeurIPS 2021 · better on 2 of 2 shared results)
Matched by method name in results tables read by AI. Authors' own tables tend to favour the newer method, and a comparison can be missed when a method is renamed.
Closest papers by meaning
- Beyond Linear Approximations: A Novel Pruning Approach for Attention Matrix (ICLR 2025 · similarity 0.85)
- Positional LSH: Binary Block Matrix Approximation for Attention with Linear Biases (NeurIPS 2026 · similarity 0.85)
- MagicPIG: LSH Sampling for Efficient LLM Generation (ICLR 2025 · similarity 0.85)
- SageAttention3: Microscaling FP4 Attention for Inference and An Exploration of 8-Bit Training (NeurIPS 2025 · similarity 0.85)
- Mixture-of-Top-$k$ Attention: Efficient Attention as Scalable Fast Weights (NeurIPS 2026 · similarity 0.84)
- Revisiting the Integration of Convolution and Attention for Vision Backbone (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
