Research map / Architectures and efficiency / RNNs and normalization
Do We Always Need the Simplicity Bias? Looking for Optimal Inductive Biases in the Wild
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Meta-learns dataset-specific spline-parametrized activation functions via bi-level episodic optimization to show that ReLU's simplicity bias is suboptimal on regression, tabular, shortcut-learning and grokking tasks but near-optimal for image classification.
Problem
Neural networks default to a simplicity bias largely induced by ReLU activations, which may be detrimental on some tasks; the paper asks when this bias is inadequate and what optimal inductive biases look like.
Approach
Parametrize activations as unconstrained linear splines and meta-learn them with a bi-level, episodic loop that optimizes generalization of reusable activations, enabling discovery of novel activations with higher-complexity inductive biases where ReLUs are suboptimal.
Key results
- On image classification, learned activations match ReLUs closely, and ReLU-initialized optimization converges to a GeLU-like function.
- On tabular data, learned input activations eliminate the gap to boosted trees on 16 datasets and sometimes surpass them.
- On modular addition grokking, the learned-activation model converges about 10x faster than ReLUs and eliminates delayed generalization.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace ReLU with a dataset-specific learned activation | MLPs on image classification datasets · MNIST, FASHION-MNIST, SVHN, CIFAR-10 | accuracy | No clear effect |
| Replace ReLU with a learned activation initialized from zeros | MLPs on image regression tasks · MNIST, FASHION-MNIST, SVHN, CIFAR-10 regression | accuracy | Helped |
| Replace ReLU with tuned-prefactor TanH or learned activations | MLPs on tabular datasets · 16 tabular datasets | accuracy | Helped |
| Add learned input activation functions | MLPs on tabular datasets · 16 tabular datasets | accuracy | Helped |
| Replace ReLU with an activation optimized for a selected input feature | MLPs on MNIST/CIFAR collages · MNIST/CIFAR shortcut-learning task | accuracy | Mixed |
| Replace ReLU with a task-specific learned activation | MLPs on algorithmic tasks · Algorithmic tasks | test accuracy | Helped |
| Replace ReLU with a task-specific learned activation | MLPs on addition mod 27 · Addition mod 27 | convergence speed: ∼10× faster | Helped |
| Learn one activation jointly across related tasks rather than using ReLU | MLPs on algorithmic tasks · 22 algorithmic tasks | accuracy: 19.9% vs. 61.5% | Helped |
| Learn an activation jointly across tasks rather than task-specific activations | MLPs on algorithmic tasks · 22 algorithmic tasks | accuracy: 54.0% to 61.5% | Helped |
| Initialize learned activation parameters as zeros instead of mimicking ReLU (ablation) | MLPs on image regression tasks · Image regression | accuracy | Helped |
| Use a separate learned activation per layer instead of sharing one activation (ablation) | MLPs on image classification and regression · Image datasets | accuracy | No clear effect |
| Use learned activations in networks with fewer layers than the ReLU networks (ablation) | MLPs on tabular datasets · Tabular datasets | accuracy | Helped |
Surprising findings
- Learned activations initialized from zeros rediscover GeLU-like smooth functions on image classification, suggesting ReLUs are near a local optimum there.
- Some models with learned activations overfit with prolonged training while ReLU networks keep high accuracy once they grok.
Limitations
- Experiments use simple MLPs; findings may not transfer to state-of-the-art architectures.
- Complexity measured with a total-variation proxy rather than a complete characterization of inductive biases.
- Learned activations on shortcut learning do not fully avoid shortcut reliance and training dynamics matter.
Experimental setup
Datasets: MNIST Fashion-MNIST SVHN CIFAR-10 16 tabular datasets from Grinsztajn et al. MNIST/CIFAR collages algorithmic modular arithmetic tasks (grokking) COVERTYPE ELECTRICITY
Benchmarks: MNIST Fashion-MNIST SVHN CIFAR-10 16 tabular datasets (Grinsztajn et al.) MNIST/CIFAR collages algorithmic grokking tasks
Metrics: test accuracy total variation complexity convergence steps
Baselines: ReLU MLP TanH with tuned prefactor boosted decision trees k-NN linear classifier
Builds on: Teney 2024 simplicity bias from ReLU activations GeLU grokking Grinsztajn tabular benchmark
Closest papers by meaning
- Simplicity Bias of Two-Layer Networks beyond Linearly Separable Data (ICML 2024 · similarity 0.89)
- The Rich and the Simple: On the Implicit Bias of Adam and SGD (NeurIPS 2025 · similarity 0.88)
- Simplicity Bias in Overparameterized Machine Learning (AAAI 2024 · similarity 0.88)
- Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks (ICML 2025 · similarity 0.87)
- Simplicity Bias via Global Convergence of Sharpness Minimization (ICML 2024 · similarity 0.86)
- Initial Guessing Bias: How Untrained Networks Favor Some Classes (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.