Research map / Optimization / Implicit bias and generalization
Neural Thermodynamics: Entropic Forces in Deep and Universal Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an entropic-force theory of SGD (effective entropy from discretization and gradient noise) showing it breaks continuous parameter symmetries, yielding gradient-balance equipartition results that explain universal representation alignment (a proof of a perfect Platonic Representation Hypothesis in deep linear nets) and progressive sharpening.
Problem
Understanding emergent phenomena in deep learning (progressive sharpening/flattening, universal representation alignment across models) that loss minimization alone does not explain, from the learning dynamics of SGD.
Approach
Derives an entropic loss F = expected loss + regularization + effective entropy from discretization/noise of SGD; shows this entropy breaks continuous parameter symmetries while preserving discrete orthogonal ones, yielding gradient-balance (equipartition) theorems across layers and neurons that drive universal representation alignment and sharpness dynamics.
Key results
- Two independently trained 6-layer linear/tanh networks on randomly transformed MNIST become perfectly aligned across every pair of layers (theory predicts perfect alignment under gamma=0, eta to 0).
- Theory predicts sharpness of two-layer linear nets can be arbitrarily large under label-noise imbalance while the minimum sharpness is independent of noise covariance.
- Layer and neuron gradient balance quantities approach zero during training of ReLU and linear teacher-student networks, correlated with entropy rather than loss.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase balance in label noise | two-layer linear network trained with SGD · linear regression task | sharpness (η λ_max) | Helped |
| increase imbalance in label noise | two-hidden-layer ReLU network · linear regression task | sharpness at convergence | Hurt |
| decrease the learning rate | ResNet18 trained on CIFAR-10 · CIFAR-10 training | entropy | Helped |
| increase weight decay (ablation) | independently trained student networks · teacher-student representation alignment | representation alignment | Hurt |
| vary batch size (ablation) | independently trained student networks · teacher-student representation alignment | representation alignment | No clear effect |
| train with weight balance instead of gradient balance (η=0, γ→0+) (ablation) | deep linear network · universal representation learning | representation alignment | Hurt |
| use a polynomial activation with degree d≠1 (ablation) | polynomial network · gradient balance | gradient fluctuation across layers | Hurt |
Surprising findings
- Weight decay destroys universal alignment, and the perfect-alignment result holds for any deep linear network regardless of capacity, suggesting symmetry alone drives universality.
- Entropy term can increase at the first learning rate drop in ResNet18 training, which the theory attributes to the entropic loss.
Limitations
- Analysis restricted to problems with explicit (exact) parameter symmetries; approximate symmetries not covered.
- First-order expansion in learning rate; higher-order corrections matter near edge of stability.
- Adam and generic matrix learning rates break orthogonal invariances, so results do not transfer directly.
Experimental setup
Datasets: MNIST CIFAR-10 ImageNet
Benchmarks: MNIST CIFAR-10
Metrics: layer balance neuron balance cosine representation alignment CKA sharpness (trace of Hessian) test accuracy
Baselines: Platonic Representation Hypothesis
Builds on: modified loss (implicit regularization of SGD) exponential symmetry Platonic Representation Hypothesis edge of stability
Closest papers by meaning
- Parameter Symmetry and Noise Equilibrium of Stochastic Gradient Descent (NeurIPS 2024 · similarity 0.86)
- Emergence and scaling laws in SGD learning of shallow neural networks (NeurIPS 2025 · similarity 0.86)
- Symmetries in Overparametrized Neural Networks: A Mean Field View (NeurIPS 2024 · similarity 0.86)
- Parameter symmetries determine representational geometry in overparameterized nonlinear networks (NeurIPS 2026 · similarity 0.86)
- Formation of Representations in Neural Networks (ICLR 2025 · similarity 0.85)
- Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.