Research map / Architectures and efficiency / Attention and transformers
MonarchAttention: Zero-Shot Conversion to Fast, Hardware-Aware Structured Attention
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MonarchAttention, a zero-shot, hardware-efficient sub-quadratic attention substitute that projects softmax attention onto Monarch matrices via alternating maximization of the softmax variational objective.
Problem
Softmax attention has quadratic complexity in sequence length, and existing sub-quadratic alternatives either require retraining or fail to deliver wall-clock speedups on modern GPUs.
Approach
Approximate softmax attention by a Monarch matrix found by alternating maximization of the softmax variational objective (entropy-regularized linear objective) over block rank-one factors, which reduces cost to Theta(N sqrt(N) d) and maps onto batched matmuls on tensor cores and FlashAttention-like fused kernels.
Key results
- On an A40 GPU, MonarchAttention is up to 8.2x faster than FlashAttention-2 at N=16384 and up to 1.4x faster at N=256 with a fully-fused kernel.
- On ViT-B ImageNet, MonarchAttention loses about 5% accuracy while cutting attention FLOPs by 80%, versus baselines that degrade more.
- On DiT-XL with the first half of layers replaced, MonarchAttention reaches FID 0.39 and sFID 0.66 against softmax reference images, versus 8.17 FID for Nyströmformer at similar FLOPs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace all DiT attention layers with MonarchAttention instead of Nyströmformer | DiT-XL · ImageNet image generation | FID: 5.97 -> 2.82 | Helped |
| Replace all DiT attention layers with MonarchAttention instead of Nyströmformer | DiT-XL · ImageNet image generation | sFID: 13.47 -> 5.09 | Helped |
| Replace BART encoder attention with zero-shot MonarchAttention instead of zero-shot Nyströmformer | BART-B · BookSum-chapters, 8192 tokens | ROUGE-1: 32.86 -> 34.60 | Helped |
| Fine-tune after swapping in MonarchAttention | BART-B · BookSum-chapters, 8192 tokens | ROUGE-1: 34.60 -> 35.38 | Helped |
| Replace attention in BART with MonarchAttention rather than softmax | BART-B · BookSum-chapters, 8192 tokens | ROUGE-1: 35.57 -> 34.60 | Hurt |
| Increase alternating-maximization steps from 0 to 1 (ablation) | MonarchAttention · ViT-B, layer 0 head 5 | variational objective: 418 -> 937 | Helped |
| Increase alternating-maximization steps from 1 to 2 (ablation) | MonarchAttention · ViT-B, layer 0 head 5 | variational objective: 937 -> 940 | Helped |
| Increase alternating-maximization steps from 2 to 3 (ablation) | MonarchAttention · ViT-B, layer 0 head 5 | variational objective: 940 -> 940 | No clear effect |
| Replace RoBERTa layer 6 attention rather than an initial or final layer (ablation) | RoBERTa-B · SQuAD1.1 | F1: 86.3 F1 | Hurt |
| Replace the fourth quarter rather than earlier quarters of DiT attention (ablation) | DiT-XL · ImageNet image generation | FID: 0.07, 0.21, 0.34, 1.35 | Hurt |
| Use MonarchAttention instead of softmax attention | Transformers · Attention runtime, N=256 | speedup: 1.4× | Helped |
| Use MonarchAttention instead of FlashAttention-2 | Transformers · Attention runtime, N=4K | speedup: 4.5× | Helped |
Surprising findings
- Replacing the first half of DiT attention layers with MonarchAttention yields nearly identical images to softmax (FID 0.39), while the final quarter is the most sensitive to replacement.
- MonarchAttention on BART summarization at N=8192 outperforms softmax attention at N=2048 on ROUGE-1 and ROUGE-L with fewer FLOPs.
Limitations
- Cannot be applied directly to autoregressive decoding since there is no full attention matrix to approximate; causal masking is not efficiently incorporated.
- Uniform block sizes allocate the same parameters to every block, which may under-approximate regions needing finer detail.
- Layer sensitivity varies: middle layers of RoBERTa and final DiT layers degrade more when replaced.
Experimental setup
Datasets: ImageNet-1K ImageNet-21K SQuAD1.1 BookSum-chapters ImageNet Actor graph dataset
Benchmarks: ImageNet-1K SQuAD1.1 BookSum-chapters ImageNet FID (DiT generation) Actor node classification
Metrics: top-5 accuracy F1 ROUGE-1 ROUGE-L FID sFID runtime speed-up attention FLOPs
Baselines: linear-attention performer cosformer nystromformer FlashAttention-2
Builds on: FlashAttention-2 Monarch matrices Performer Nyströmformer cosFormer
Closest papers by meaning
- MonarchRT: Efficient Attention for Real-Time Video Generation (NeurIPS 2026 · similarity 0.90)
- XAttention: Block Sparse Attention with Antidiagonal Scoring (ICML 2025 · similarity 0.89)
- FlashMask: Efficient and Rich Mask Extension of FlashAttention (ICLR 2025 · similarity 0.89)
- CAT: Circular-Convolutional Attention for Sub-Quadratic Transformers (NeurIPS 2025 · similarity 0.88)
- kNN Attention Demystified: A Theoretical Exploration for Scalable Transformers (ICLR 2025 · similarity 0.88)
- Gated Linear Attention Transformers with Hardware-Efficient Training (ICML 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.