Research map / Optimization / Implicit bias and generalization

Convergence of Near-Linear Width ReLU Networks with Unbalanced Initialization

NeurIPS 2026 · Optimization › Implicit bias and generalization · approach: Shallow ReLU network convergence

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows gradient descent convergence for two-layer ReLU networks at near-linear width under unbalanced initialization, addressing the ReLU-induced kernel shift in the NTK regime.

Problem

Two-layer ReLU networks in the NTK regime require substantially larger width than linear for convergence, because activation-pattern changes during training induce kernel shift.

Approach

Uses unbalanced initialization of the network to control the ReLU-induced kernel shift so that convergence holds at near-linear width, closing the gap with random-feature and conjugate kernel analyses.

Key results

• Establishes convergence of two-layer ReLU networks in the NTK regime at near-linear width under unbalanced initialization (abstract truncated; specific bounds not available).

Limitations

• Full text unavailable; only abstract used, so specific width bounds and assumptions are unknown.

Experimental setup

Builds on: neural tangent kernel conjugate kernel regime analysis

Closest papers by meaning

• Stable Minima Cannot Overfit in Univariate ReLU Networks: Generalization by Large Step Sizes (NeurIPS 2024 · similarity 0.89)
• Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training (ICML 2025 · similarity 0.89)
• Better NTK Conditioning: A Free Lunch from (ReLU) Nonlinear Activation in Wide Neural Networks (NeurIPS 2025 · similarity 0.89)
• Early Neuron Alignment in Two-layer ReLU Networks with Small Initialization (ICLR 2024 · similarity 0.89)
• Expressivity of ReLU-Networks under Convex Relaxations (ICLR 2024 · similarity 0.88)
• Simplicity Bias and Optimization Threshold in Two-Layer ReLU Networks (ICML 2025 · similarity 0.88)