AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture

Daniel Y Fu, Simran Arora, Jessica Grogan, Isys Johnson, Sabri Eyuboglu, Armin W Thomas, Benjamin Frederick Spector, Michael Poli, Atri Rudra, Christopher Re

NeurIPS 2023 · Architectures and efficiency › Attention and transformers · approach: State space and recurrent sequence models · 13 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Replace BERT attention and MLP layers with M2 layers at 27% fewer parametersBERT-base · GLUE averageGLUE Score: 79.6 -> 79.9; -27% parametersImproved
Replace BERT attention and MLP layers with M2 layers at parameter-matched sizeBERT-base · GLUE averageGLUE Score: 79.6 -> 80.9; +1.3Improved
Replace BERT attention and MLP layers with M2 layers at 24% fewer parametersBERT-large · GLUE averageGLUE Score: 82.1 -> 82.2; -24% parametersImproved
Replace BERT attention and MLP layers with M2 layers at parameter-matched sizeBERT-large · GLUE averageGLUE Score: 82.1 -> 82.8; +0.7Improved
Use M2 layers instead of standard BERT layersBERT-base · BERT-base inference, sequence length 4096throughput (tokens/ms): 39.0 -> 353.9; 9.1×Improved
Replace attention and MLP blocks with M2 operatorsViT-b · ImageNet-1kTop-1 accuracy (%): 78.5 -> 79.5; 1% higher accuracyImproved
Use causal Monarch parameterization and remove MLP layersGPT-style language model · The PILE, 15B tokens, 360M modelperplexity: 9.1 -> 9.0Improved
Make sequence-mixer Monarch matrices learnable instead of fixed (ablation)M2 · sCIFARaccuracy: 91.0 -> 92.5; 1.5 pointsImproved
Use M2-BERT instead of BERT for CPU inference at sequence length 512BERT-base · CPU inference, sequence length 512speedup: 0.6×Hurt
Use M2-BERT instead of BERT for CPU inference at sequence length 8192BERT-base · CPU inference, sequence length 8192speedup: 6.5×Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy