Research map / Architectures and efficiency / RNNs and normalization
Initial Guessing Bias: How Untrained Networks Favor Some Classes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically and empirically shows that untrained DNN architecture and input preprocessing can make an initialized network assign most or all inputs to a single class ("Initial Guessing Bias"), via node symmetry breaking in infinite-width MLPs with random data.
Problem
Untrained neural networks can systematically assign predictions to one class at initialization even without explicit biases; the role of architecture and data preprocessing in this initial predictive bias is unexplored.
Approach
Defines Initial Guessing Bias (IGB) and shows, via an infinite-width analysis that averages over data first and then weights, that node symmetry breaking caused by nonlinearities (ReLU, max pooling), input offsets, and depth produces class imbalance at initialization; the variance ratio gamma quantifies IGB.
Key results
- Theory shows that for ReLU MLPs with Kaiming init and zero biases, class-fraction distributions stay wide, while linear networks concentrate at 1/2 (IGB absent).
- Max pooling with large kernel size and input offset can drive the variance ratio to infinity, i.e. strong IGB where all data go to one class.
- Accuracy upper bound: Accuracy(t) <= 1 - (max_c fraction_c - 1/K), so strong IGB limits achievable accuracy early in training.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ReLU instead of a linear activation | Single hidden-layer MLP at initialization · Gaussian random data | spread of the class 0 guess fraction across initializations | Mixed |
| Add max pooling after ReLU | Single hidden-layer MLP at initialization · Gaussian random data | concentration of predictions on one class | Hurt |
| Shift the input distribution with a nonzero offset | ReLU MLP at initialization · Centered Gaussian data before preprocessing | initial guessing bias | Helped |
| Increase max-pooling kernel size | ReLU MLP at initialization · Centered Gaussian data | initial guessing bias | Helped |
| Increase network depth | MLP at initialization · Centered Gaussian data without pooling or preprocessing offset | initial guessing bias | Mixed |
| Use an antisymmetric activation with zero-centered data | MLP at initialization · Zero-centered data | initial guessing bias | Helped |
| Shift the ReLU function appropriately | MLP at initialization · Zero-centered data | initial guessing bias | Helped |
| Use structured real data rather than unstructured random data | Untrained models across practical architectures · Real datasets | initial guessing bias | Hurt |
| Increase the level of initial guessing bias | Models during training · Balanced datasets | time to reabsorb the bias | Hurt |
| Increase the level of initial guessing bias | Models during training · Balanced datasets | training performance | Hurt |
Surprising findings
- Symmetry between classes is preserved in ensemble average, yet a single realization is strongly biased, so the observable does not self-average.
- Network depth cannot initiate IGB but amplifies it when already present.
Limitations
- Main theory uses infinite-width limit and i.i.d. random data; finite-size and real-data effects are treated empirically.
- Impact of IGB on final generalization is not fully characterized and is described as data- and architecture-dependent.
Experimental setup
Datasets: random i.i.d. Gaussian data real image datasets (binary and multi-class)
Metrics: fraction of points classified per class accuracy variance ratio gamma
Baselines: MLP-Mixer linear activation network
Builds on: Kaiming initialization MLP-Mixer
Closest papers by meaning
- When Bias Helps Learning: Bridging Initial Prejudice and Trainability (ICLR 2026 · similarity 0.94)
- Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks (ICML 2025 · similarity 0.87)
- Feature Averaging: An Implicit Bias of Gradient Descent Leading to Non-Robustness in Neural Networks (ICLR 2025 · similarity 0.86)
- Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networks (ICML 2024 · similarity 0.86)
- If You Want to Be Robust, Be Wary of Initialization (NeurIPS 2024 · similarity 0.86)
- Simplicity Bias in Overparameterized Machine Learning (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.