Research map / Architectures and efficiency / RNNs and normalization
When Representations Align: Universality in Representation Learning Dynamics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives a two-point effective theory of representation learning dynamics that assumes arbitrary smooth encoder/decoder maps, and shows it matches training trajectories across many architectures and reproduces rich/lazy-like structured representations.
Problem
Understanding representation learning dynamics in a way that is universal across architectures, since exact analyses of learning dynamics are architecture-specific and do not scale to modern networks.
Approach
Model the interaction of two datapoints' representations by linearizing the smooth encoder and decoder around their midpoint and applying gradient descent directly to the linearization parameters (assuming expressivity), yielding a self-contained 3-dimensional ODE system with two effective learning rates that is architecture-agnostic.
Key results
- Effective theory with two fitted constants matches representational distance, prediction difference and alignment curves for a 20-layer leaky ReLU network and ten architecture variants.
- Final representational distance from theory matches learned distances for two-point experiments across varied input and output distances after fitting learning rates at one fixed setting.
- On MNIST at small initial weights, theory vs. experiment pairwise distances reach Pearson correlation 0.703 (0.512 excluding diagonal blocks).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Model two-point representation dynamics with the effective theory | Deep networks with varying architectures and activation functions · Two-point dataset | Representation dynamics fit | Helped |
| Increase the initial weight scale (ablation) | Default 20-layer network · Two-point dataset | Plateau-like behavior in representational distance and loss | Mixed |
| Use very high initial weights (ablation) | Default 20-layer network · Two-point dataset | Agreement with effective theory | Hurt |
| Use a 2-layer architecture instead of the default 20-layer network (ablation) | XOR task · XOR | Agreement of learned representation with theory | Hurt |
| Use small initial weights (ablation) | Deep network · XOR | Merging of representations for datapoints with the same target | Helped |
| Use small initial weights rather than large initial weights (ablation) | 5-layer tanh fully connected network · Orthogonal feature dataset | Collapse along task-irrelevant feature direction | Helped |
| Apply exponential weighing to the theory distances | Effective theory · MNIST pairwise distances | Pearson correlation with experiment: 0.703; 0.512 when not factoring in the diagonal blocks | Helped |
| Compare theory predictions at later rather than earlier hidden layers (ablation) | Default 20-layer network · Two-point dataset | Agreement with effective theory | Helped |
| Use a 5-layer tanh architecture for the feature-collapse experiment (ablation) | Representation learning experiment · Orthogonal feature dataset | Task-irrelevant feature collapse | Helped |
Surprising findings
- The theory fails to predict pairwise merging in a 2-layer XOR network at small weights, while matching the 20-layer network.
Limitations
- No clear path to extending the two-point theory to large, complex datasets; multi-point effects are only heuristically tested on MNIST.
- Ignores architectural inductive biases that affect learned representations.
- Theory is accurate only in later hidden layers; fails for the 2-layer XOR network.
Experimental setup
Datasets: two-point synthetic dataset XOR Gaussian blobs orthogonal feature dataset MNIST
Benchmarks: two-point learning dynamics XOR representational structure orthogonal feature collapse MNIST pairwise distances
Metrics: representational distance prediction difference output alignment training loss Pearson correlation
Baselines: neural tangent kernel two-layer linear network
Builds on: neural tangent kernel local elasticity rich and lazy learning
Closest papers by meaning
- From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks (ICLR 2025 · similarity 0.89)
- Formation of Representations in Neural Networks (ICLR 2025 · similarity 0.88)
- How are linear representations learned? Exact solutions to the dynamics of abstraction (NeurIPS 2026 · similarity 0.88)
- Tracing Representation Progression: Analyzing and Enhancing Layer-Wise Similarity (ICLR 2025 · similarity 0.88)
- Task structure and nonlinearity jointly determine learned representational geometry (ICLR 2024 · similarity 0.88)
- Three Mechanisms of Feature Learning in a Linear Network (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.