Research map / Architectures and efficiency / RNNs and normalization
Understanding MLP-Mixer as a wide and sparse MLP
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that MLP-Mixer is effectively a wide, sparse MLP with Kronecker-product weights, links it to implicit L1 regularization and Monarch matrices, and argues that wider-sparser width-fixed-connections settings improve accuracy.
Problem
Why and how the MLP-Mixer outperforms conventional MLPs remains unexplained; the paper asks which architectural inductive biases (sparsity, width) drive its performance.
Approach
Vectorizing mixing layers shows that a Mixer equals a much wider MLP whose weights are Kronecker products (sparse, structured); this links Mixers to implicit L1 regularization, Monarch matrices, and fixed-connection sparse-weight MLPs, and suggests balancing token and channel sizes (S≈C) to maximize effective width under a fixed connection budget.
Key results
- On CIFAR-10 and CIFAR-100, the wider Mixer-SS-W reaches 12.07% and 38.13% test error versus 14.81% and 40.44% for β-LASSO at similar connection counts.
- On ImageNet-1k, the wider Mixer-B-W (S=256, C=588) achieves 23.26% top-1 error versus 23.56% for Mixer-B/16 under the same Ω.
- Normal and random-permuted Mixers show similar error-vs-width trends, and the test error is minimized around C=S under fixed connections.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Widening Mixer by setting S and C closer under fixed total connections (Mixer-SS/8 to Mixer-SS-W) | MLP-Mixer on CIFAR-10 · CIFAR-10 | test error (%): 15.91 to 12.07 | Helped |
| Widening Mixer by setting S and C closer under fixed connections (Mixer-SS/8 to Mixer-SS-W) | MLP-Mixer on CIFAR-100 · CIFAR-100 | test error (%): 44.24 to 38.13 | Helped |
| Setting S and C closer under fixed Omega (Mixer-B/16 to Mixer-B-W) | MLP-Mixer on ImageNet-1k · ImageNet-1k | top-1 test error (%): 23.56 to 23.26 | Helped |
| Sparse-weight MLP vs wider Mixer: test error at too-wide widths (gamma m around 8000) | SW-MLP · fixed Omega=2^19 synthetic width sweep (CIFAR-10 style) | test error | Hurt |
| Random permutation of Mixer (RP-Mixer) vs normal Mixer (ablation) | S-Mixer / MLP-Mixer · CIFAR-10, CIFAR-100, STL-10, ImageNet-1k | test error | Mixed |
| Maximizing effective width under fixed connections (test error minimized around C=S) (ablation) | S-Mixer, RP S-Mixer, MLP-Mixer · CIFAR-10, CIFAR-100, STL-10, ImageNet-1k | test error | Helped |
| Random-permuted mixing weights (RP) in SW-MLP-like alternative with large width | SW-MLP alternative · largest-width singular value of masked weights | max singular value | Hurt |
| Sparse-weight (static mask) MLP width increase under fixed connections | SW-MLP · MNIST/CIFAR sweep, Omega=2^19 | test error | Helped |
Surprising findings
- Random-permuted Mixers, whose weights look like random sparse matrices, can match or beat normal Mixers when depth increases.
Limitations
- The implicit L1 regularization result is proven only for the linear S-Mixer and the bound is an inequality with normalization choices.
- Sparse-weight MLP comparisons are limited to specific sizes because static-mask SW-MLPs need m×m matrices, so very large widths are not directly compared.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 STL-10 MNIST ImageNet-1k
Benchmarks: CIFAR-10 CIFAR-100 STL-10 ImageNet-1k MNIST
Metrics: test error centered kernel alignment singular value spectrum
Baselines: MLP-Mixer (Mixer-B/16) sparse-weight MLP (SW-MLP) β-LASSO dense MLP
Builds on: MLP-Mixer Monarch matrices sparse-weight MLP fixed connectivity (Golubeva et al. 2021) centered kernel alignment
Closest papers by meaning
- Random Sparse Lifts: Construction, Analysis and Convergence of finite sparse networks (ICLR 2024 · similarity 0.85)
- Position: A Theory of Deep Learning Must Include Compositional Sparsity (ICML 2025 · similarity 0.84)
- Minimum Width for Deep, Narrow MLP: A Diffeomorphism Approach (NeurIPS 2025 · similarity 0.83)
- Global Minimizers of $\ell^p$-Regularized Objectives Yield the Sparsest ReLU Neural Networks (NeurIPS 2025 · similarity 0.83)
- An Infinite-Width Analysis on the Jacobian-Regularised Training of a Neural Network (ICML 2024 · similarity 0.83)
- A Single-Step, Sharpness-Aware Minimization is All You Need to Achieve Efficient and Accurate Sparse Training (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.