Research map / Architectures and efficiency / RNNs and normalization
Neural Redshift: Random Networks are not Random Functions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that untrained random-weight networks already have architecture-dependent complexity biases (simplicity bias for ReLU-like activations, residual connections, layer norm), and that these biases predict generalization and shortcut learning in trained MLPs and random-weight transformers.
Problem
Explaining neural network generalization independently of gradient-based optimization, since existing explanations (implicit bias of GD) fail to account for gradient-free training and the simplicity bias seen in untrained networks.
Approach
Characterize architectural inductive biases by sampling random weights uniformly and measuring the complexity (Fourier frequency, polynomial order, LZ compressibility) of the resulting functions; the parametrization itself (ReLUs, residuals, layer norm) sets a preferred complexity, which then predicts trained-model behavior.
Key results
- ReLU, GELU and Swish MLPs at random initialization have consistently low complexity across depth and weight magnitude, while TanH, Gaussian and sine activations show complexity that grows with depth and weight scale.
- Adding layer normalization removes weight-magnitude sensitivity for all activations, and residual connections push preferred complexity to low levels regardless of depth.
- On modulo addition, ReLU MLPs only solve the low-frequency version; Gaussian MLPs with a tuned prefactor solve all three versions, with the optimal prefactor shifting upward with task complexity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ReLU-like activations instead of other activations | Random MLPs · Random-weight functions | Fourier complexity | Helped |
| Increase depth (ablation) | Random ReLU MLPs · Random-weight functions | Fourier complexity | No clear effect |
| Increase weight magnitude (ablation) | Random ReLU MLPs · Random-weight functions | Fourier complexity | No clear effect |
| Replace ReLU with TanH, Gaussian, or sine activations (ablation) | Random MLPs · Random-weight functions | Fourier complexity | Mixed |
| Increase network width (ablation) | Random MLPs · Random-weight functions | Complexity | No clear effect |
| Add layer normalization before each activation (ablation) | Random MLPs · Random-weight functions | Complexity sensitivity to weight magnitude | Helped |
| Add residual connections (ablation) | Random MLPs · Random-weight functions | Complexity | Helped |
| Add multiplicative interactions (gating) (ablation) | Random MLPs · Random-weight functions | Complexity | Hurt |
| Use an inverse-Fourier-transform architecture with a uniform prior over frequencies (ablation) | Unbiased model · Random-weight functions | Fourier complexity | Hurt |
| Use a ReLU MLP instead of a complexity-modulated architecture | MLPs on modulo addition · High-frequency modulo tasks | Test accuracy | Hurt |
| Tune a multiplicative prefactor before TanH, Gaussian, or sine activations (ablation) | MLPs on modulo addition · Modulo addition tasks | Test accuracy | Helped |
| Increase the activation prefactor (ablation) | MLPs on Colored-MNIST · Colored-MNIST digit test set | Test accuracy | Mixed |
Surprising findings
- Width has no impact on the complexity bias of random networks.
- Random-weight networks implement simple functions by default, so the Chiang et al. loss-landscape result is attributed to parametrization rather than the loss function.
Limitations
- Experiments mostly use small models and data to allow 2D visualizations and Fourier computations.
- Analysis is empirical and not theoretical.
- Results do not rule out a role for gradient descent, which can navigate to complex functions.
Experimental setup
Datasets: modulo addition (synthetic) Colored-MNIST-style regression (MNIST digits with color band)
Benchmarks: modulo addition (3 frequency versions) Colored-MNIST-style shortcut learning task
Metrics: Fourier frequency complexity Chebyshev/Legendre polynomial order complexity Lempel-Ziv compressibility test accuracy
Baselines: ReLU MLP GELU MLP TanH MLP Gaussian MLP sine MLP untrained GPT-2
Builds on: simplicity bias (Goldblum et al.) spectral bias Loss landscapes Are All You Need (Chiang et al.) Glorot initialization
Closest papers by meaning
- How Uniform Random Weights Induce Non-uniform Bias: Typical Interpolating Neural Networks Generalize with Narrow Teachers (ICML 2024 · similarity 0.89)
- Simplicity Bias in Overparameterized Machine Learning (AAAI 2024 · similarity 0.86)
- Position: Deep Learning is Not So Mysterious or Different (ICML 2025 · similarity 0.86)
- Initial Guessing Bias: How Untrained Networks Favor Some Classes (ICML 2024 · similarity 0.86)
- Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networks (ICML 2024 · similarity 0.86)
- Simplicity Bias of Two-Layer Networks beyond Linearly Separable Data (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.