Research map / Optimization / Implicit bias and generalization
An Initial Alignment between Neural Network and Target is Needed for Gradient Descent to Learn
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a target with low initial alignment (3-parity or 9-staircase rather than majority vote) | 2-layer fully connected ReLU network with normalized Gaussian initialization · Boolean functions, input dimension 100 | generalization accuracy | Hurt |
| Have negligible INAL between the target and the expressive activation at initialization | Polynomial-size fully connected networks trained by noisy gradient descent · N-extension of Boolean targets under uniform inputs | weak learnability: CP(orb(\overline{f_n}), \mathcal{U}^N)=O(n^{-\frac{\epsilon}{1+\epsilon}(c-1)}) | Hurt |
| Vary the target task, producing different initial alignments | CNN with 1 VGG block and ReLU · CIFAR binary classification: cat/dog, bird/deer, frog/truck | generalization accuracy | Mixed |
Closest papers by meaning
- Early Neuron Alignment in Two-layer ReLU Networks with Small Initialization (ICLR 2024 · similarity 0.85)
- Revisiting Zeroth-Order Optimization: Minimum-Variance Two-Point Estimators and Directionally Aligned Perturbations (ICLR 2025 · similarity 0.84)
- Learning High-Degree Parities: The Crucial Role of the Initialization (ICLR 2025 · similarity 0.84)
- Mirror Descent Beyond Euclidean Stability: An Exponential Separation in Initialization Sensitivity (NeurIPS 2026 · similarity 0.84)
- Large Stepsize Gradient Descent for Non-Homogeneous Two-Layer Networks: Margin Improvement and Fast Optimization (NeurIPS 2024 · similarity 0.84)
- Transformers Implement Functional Gradient Descent to Learn Non-Linear Functions In Context (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.