Research map / Architectures and efficiency / Attention and transformers
Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture
Daniel Y Fu, Simran Arora, Jessica Grogan, Isys Johnson, Sabri Eyuboglu, Armin W Thomas, Benjamin Frederick Spector, Michael Poli, Atri Rudra, Christopher Re
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace BERT attention and MLP layers with M2 layers at 27% fewer parameters | BERT-base · GLUE average | GLUE Score: 79.6 -> 79.9; -27% parameters | Improved |
| Replace BERT attention and MLP layers with M2 layers at parameter-matched size | BERT-base · GLUE average | GLUE Score: 79.6 -> 80.9; +1.3 | Improved |
| Replace BERT attention and MLP layers with M2 layers at 24% fewer parameters | BERT-large · GLUE average | GLUE Score: 82.1 -> 82.2; -24% parameters | Improved |
| Replace BERT attention and MLP layers with M2 layers at parameter-matched size | BERT-large · GLUE average | GLUE Score: 82.1 -> 82.8; +0.7 | Improved |
| Use M2 layers instead of standard BERT layers | BERT-base · BERT-base inference, sequence length 4096 | throughput (tokens/ms): 39.0 -> 353.9; 9.1× | Improved |
| Replace attention and MLP blocks with M2 operators | ViT-b · ImageNet-1k | Top-1 accuracy (%): 78.5 -> 79.5; 1% higher accuracy | Improved |
| Use causal Monarch parameterization and remove MLP layers | GPT-style language model · The PILE, 15B tokens, 360M model | perplexity: 9.1 -> 9.0 | Improved |
| Make sequence-mixer Monarch matrices learnable instead of fixed (ablation) | M2 · sCIFAR | accuracy: 91.0 -> 92.5; 1.5 points | Improved |
| Use M2-BERT instead of BERT for CPU inference at sequence length 512 | BERT-base · CPU inference, sequence length 512 | speedup: 0.6× | Hurt |
| Use M2-BERT instead of BERT for CPU inference at sequence length 8192 | BERT-base · CPU inference, sequence length 8192 | speedup: 6.5× | Improved |
Closest papers by meaning
- Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers (NeurIPS 2024 · similarity 0.88)
- MonarchAttention: Zero-Shot Conversion to Fast, Hardware-Aware Structured Attention (NeurIPS 2025 · similarity 0.87)
- Understanding MLP-Mixer as a wide and sparse MLP (ICML 2024 · similarity 0.85)
- Transformer Block Coupling and its Correlation with Generalization in LLMs (ICLR 2025 · similarity 0.84)
- Flash Inference: Near Linear Time Inference for Long Convolution Sequence Models and Beyond (ICLR 2025 · similarity 0.84)
- Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
