Research map / Architectures and efficiency / RNNs and normalization
Scaling Down Deep Learning with MNIST-1D
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces MNIST-1D, a procedurally generated 40-dimensional 1D digit-like dataset, and shows it can reproduce science-of-deep-learning phenomena (inductive biases, lottery tickets, double descent, metalearning, SSL, pooling) cheaply on CPU/single GPU.
Problem
Science-of-deep-learning studies require large compute, and standard small benchmarks like MNIST are too easy, too large, and hard to vary, so they fail to separate architectures or support fast exploratory research.
Approach
A minimalist 1D procedurally generated digit dataset (40 features, 4000+1000 default samples) with tunable generation parameters (translation, noise, shear, padding) that separates model inductive biases (linear 32%, MLP 68%, CNN 94%, human 96%) and can be run in minutes on a CPU.
Key results
- Logistic regression reaches 32% on MNIST-1D versus 94% for a CNN and 96% for a human expert, while all models exceed 99% on MNIST.
- Shuffling features drops CNN accuracy from 94% to 56% and GRU from 91% to 57%, while MLP and logistic stay unchanged.
- A metalearned activation function outperforms ReLU, ELU, and Swish by over 5 percentage points test accuracy, with the whole run taking about 1 hour on CPU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use spatially ordered rather than shuffled input features (ablation) | CNN · MNIST-1D test | accuracy: 56 ± 2 -> 94 ± 2 | Helped |
| use spatially ordered rather than shuffled input features (ablation) | GRU · MNIST-1D test | accuracy: 57 ± 2 -> 91 ± 2 | Helped |
| use spatially ordered rather than shuffled input features (ablation) | MLP · MNIST-1D test | accuracy: 68 ± 2 -> 68 ± 2 | No clear effect |
| use spatially ordered rather than shuffled input features (ablation) | Logistic regression · MNIST-1D test | accuracy: 32 ± 1 -> 32 ± 1 | No clear effect |
| use a CNN rather than logistic regression | MNIST-1D classification · MNIST-1D test | accuracy: 32 ± 1 -> 94 ± 2 | Helped |
| use the learned activation function instead of common nonlinearities | MLP classifier on MNIST-1D · MNIST-1D test | accuracy: +5 percentage points | Helped |
| train with 15% label noise and increase hidden-layer width through the interpolation threshold | MLP classifier on MNIST-1D · MNIST-1D test | test error: 15%; around 50 neurons | Mixed |
| add label noise | MLP classifier on MNIST-1D · MNIST-1D test | test error: 15% label noise | Hurt |
| use pooling in low-data regimes rather than no pooling | CNN on MNIST-1D · MNIST-1D classification | sample efficiency | Helped |
| use pooling when more training data is available | CNN on MNIST-1D · MNIST-1D classification | sample efficiency | No clear effect |
| use striding instead of pooling | CNN on MNIST-1D · MNIST-1D classification | sample efficiency | No clear effect |
| use a lottery-ticket subnetwork rather than a random subnetwork at the same sparsity | MLP classifier on MNIST-1D · MNIST-1D test | accuracy: >95% sparsity | Helped |
Surprising findings
- A human expert outperforms a strong CNN (96% vs 94%) on MNIST-1D.
- Lottery tickets at over 95% sparsity performed better than the original dense network.
Limitations
- Human baseline is a single author evaluated on 500 test images, so it is a small, non-independent estimate.
- Many findings (lottery tickets, metalearning, SSL) are only shown at toy scale and their transfer to large models is untested, as the authors acknowledge.
Experimental setup
Datasets: MNIST-1D MNIST CIFAR-10 CIFAR-100 FashionMNIST
Benchmarks: MNIST-1D MNIST
Metrics: test accuracy training accuracy per-class error
Baselines: logistic regression MLP CNN GRU ReLU Swish ELU MNIST CIFAR-10
Builds on: MNIST SimCLR lottery ticket hypothesis Synthetic Petri Dish
Closest papers by meaning
- Scaling MLPs: A Tale of Inductive Bias (NeurIPS 2023 · similarity 0.84)
- Maestro: Uncovering Low-Rank Structures via Trainable Decomposition (ICML 2024 · similarity 0.84)
- Adam Reduces a Unique Form of Sharpness: Theoretical Insights Near the Minimizer Manifold (NeurIPS 2025 · similarity 0.83)
- Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation is Wasteful (NeurIPS 2025 · similarity 0.83)
- Tensor Programs VI: Feature Learning in Infinite Depth Neural Networks (ICLR 2024 · similarity 0.83)
- Understanding the Generalization of Stochastic Gradient Adam in Learning Neural Networks (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.