Research map / Optimization / Implicit bias and generalization

Diagonalizing the Softmax: Hadamard Initialization for Tractable Cross-Entropy Dynamics

NeurIPS 2026 · Optimization › Implicit bias and generalization · approach: Shallow ReLU network convergence · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that Hadamard initialization makes the softmax diagonalizable in a fixed basis for a two-layer linear UFM trained with cross-entropy, reducing gradient flow to a vector ODE on logit singular values, and proves convergence to neural collapse via a KL Lyapunov function for K=2^m.

Problem

Implicit bias of multi-class cross-entropy gradient flow in the unconstrained features model (two-layer linear network with orthogonal inputs) is unresolved; it is unknown whether gradient flow converges to neural collapse and what the trajectory looks like.

Approach

Initialize weights in a Sylvester-Hadamard basis so that the softmax of the logit matrix is diagonal in the same basis, freezing singular vectors and reducing CE dynamics to an ODE over logit singular values; a KL divergence to the uniform solution then serves as a Lyapunov function.

Key results

• KL divergence to the uniform normalized logit singular values is proven to decrease monotonically under Hadamard-initialized CE gradient flow for K=2^m, implying convergence to neural collapse.
• The L2 NC metric M is monotone for K=2 and K=4 but admits non-monotone initializations for K>=8.
• Hadamard initialization keeps residual energy at zero throughout training, and simulations match the reduced singular-value ODE.

Surprising findings

• The natural L2 distance to neural collapse is not monotone under CE gradient flow for K>=8, despite prior numerical observations of monotone convergence.
• Standard random small initialization in the experiments yields non-monotone Lyapunov function oscillations even though the L2 metric remains monotone.

Limitations

• Restricted to K=2^m with Sylvester Hadamard structure, balanced classes, unregularized, and balanced UFM with two layers.
• Hadamard initialization is a special initialization; random initialization is only approximated, with the approximation degrading as K grows.

Experimental setup

Metrics: KL divergence Lyapunov function L2 NC distance metric M cross-entropy loss residual energy outside signal subspace

Baselines: Random initialization (Gaussian) ODE reduction of singular values

Builds on: Saxe et al. exact dynamics for linear networks unconstrained features model Ji et al. implicit bias of CE gradient flow Soudry et al. max-margin implicit bias

Closest papers by meaning

• Are All Losses Created Equal: A Neural Collapse Perspective (NeurIPS 2022 · similarity 0.88)
• Learning Dynamics of Deep Matrix Factorization Beyond the Edge of Stability (ICLR 2025 · similarity 0.87)
• Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Data (NeurIPS 2025 · similarity 0.87)
• On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling (NeurIPS 2025 · similarity 0.86)
• Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer (ICML 2025 · similarity 0.86)
• Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse (ICLR 2025 · similarity 0.85)