Research map / Architectures and efficiency / RNNs and normalization
Scaling MLPs: A Tale of Inductive Bias
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add data augmentation during training (ablation) | B-MLP trained from scratch · CIFAR100 | accuracy: 30.5 -> 50.4 | Helped |
| Increase training from 1000 to 5000 epochs with data augmentation (ablation) | B-MLP · CIFAR100 | accuracy: 48.3 -> 50.4 | Helped |
| Use the inverted bottleneck architecture rather than standard MLP (ablation) | MLP pretrained on ImageNet21k · CIFAR100 | Top-1 accuracy: 68.3 -> 74.6 | Helped |
| Add test-time augmentation using random crops (ablation) | B-12/Wi-1024 + DA · ImageNet | Top-1 accuracy: 43.3 ± 0.06 -> 51.5 ± 0.1 | Helped |
| Add test-time augmentation using random crops (ablation) | B-12/Wi-1024 + DA · CIFAR10 | Top-1 accuracy: 94.2 ± 0.05 -> 95.5 ± 0.05 | Helped |
| Increase depth from 6 to 12 blocks (ablation) | B-MLP/Wi-1024 + DA · CIFAR100 | Top-1 accuracy: 76.4 ± 0.2 -> 80.0 ± 0.05 | Helped |
| Replace LayerNorm with BatchNorm (ablation) | B-6/Wi-1024 pretrained on ImageNet21k · CIFAR100 | Top-1 accuracy: 74.6 -> 73.8 | Hurt |
| Remove label smoothing (ablation) | B-6/Wi-1024 pretrained on ImageNet21k · CIFAR100 | Top-1 accuracy: 74.6 -> 72.2 | Hurt |
| Replace LION with Adam(W) (ablation) | MLP pretraining · CIFAR100 | linear-probe accuracy | Hurt |
| Increase pretraining batch size (ablation) | MLP pretrained on ImageNet21k · CIFAR100 | linear-probe accuracy | Helped |
| Turn off data augmentation during pretraining (ablation) | MLP pretrained on ImageNet21k · CIFAR10 | downstream accuracy | Hurt |
| Allocate more fixed compute to dataset size and fewer resources to parameter count (ablation) | Compute-optimal MLP · CIFAR100 | linear-evaluation error: P^{*}(C) ∝ C^{0.35}; N^{*}(C) ∝ C^{0.65} | Helped |
Closest papers by meaning
- MAXIM: Multi-Axis MLP for Image Processing (CVPR 2022 · similarity 0.88)
- When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentations (ICLR 2022 · similarity 0.88)
- MLPs Learn In-Context on Regression and Classification Tasks (ICLR 2025 · similarity 0.87)
- On the expressiveness and spectral bias of KANs (ICLR 2025 · similarity 0.86)
- LibraGrad: Balancing Gradient Flow for Universally Better Vision Transformer Attributions (CVPR 2025 · similarity 0.86)
- Role of Locality and Weight Sharing in Image-Based Tasks: A Sample Complexity Separation between CNNs, LCNs, and FCNs (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.