Research map / Label-efficient and robust learning / Recognition and long-tail
Feature Learning beyond the Lazy-Rich Dichotomy: Insights from Representational Geometry
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes manifold capacity and GLUE effective geometric measures to quantify feature learning richness and reveal learning stages and strategies beyond the lazy-rich dichotomy, across 2-layer nets, VGG-11/ResNet-18 on CIFAR, and RNNs.
Problem
Existing measures of lazy versus rich feature learning (weight changes, accuracy, NTK- and representation-label alignment) are coarse and miss the diversity of feature learning subtypes, inductive biases, and OOD behavior.
Approach
Track how task-relevant class manifolds untangle during training using mean-field manifold capacity and its effective geometric descriptors (manifold dimension, radius, center and axis alignment), rather than weight or kernel changes, to quantify feature learning and delineate learning stages and strategies.
Key results
- Manifold capacity tracks the inverse scale factor (ground truth richness) in 2-layer NNs and VGG-11 on CIFAR-10 better than weight change, accuracy, NTK-label and representation-label alignment.
- In VGG-11 on CIFAR-10, training shows four geometric stages: clustering, structuring, separating, and stabilizing.
- RNNs with small initial weight rank reach the same final capacity as large-rank ones but end with different manifold geometry (larger radius, smaller dimension vs. the opposite).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase feature-learning richness from lazy to rich training | 2-layer neural networks and VGG-11/ResNet-18 · Synthetic Gaussian clouds; CIFAR-10 | manifold capacity | Helped |
| Train in an ultra-rich regime rather than around the rich regime | Models pretrained on CIFAR-10 · CIFAR-100 test (linear probe) | accuracy | Hurt |
| Increase richness from lazy to rich training | Models pretrained on CIFAR-10 · CIFAR-10 and CIFAR-10C | test accuracy | Helped |
| Use small rather than large initial recurrent-weight rank | RNNs · NeuroGym cognitive tasks | capacity change during training | Helped |
| Use small rather than large initial recurrent-weight rank | RNNs · NeuroGym cognitive tasks | final capacity | No clear effect |
| Train in a very rich regime, first compressing radius and dimension, then increasing radius | 2-layer neural networks · Synthetic Gaussian clouds | manifold dimension and radius | Mixed |
| Increase feature-learning richness beyond the rich regime | Models pretrained on CIFAR-10 · CIFAR-100 test (linear probe) | accuracy | Hurt |
Surprising findings
- In the ultra-rich regime, CIFAR-100 linear probe accuracy drops drastically even though CIFAR-10 and CIFAR-10C accuracy keeps improving.
- RNNs with different initial weight ranks reach the same final capacity yet exhibit different manifold geometry.
Limitations
- Theoretical result restricted to one gradient step in a proportional-limit 2-layer teacher-student setting.
- Mean-field capacity is an approximation; effective measures are approximations of exact capacity.
- Experiments limited to small architectures (VGG-11, ResNet-18) and simple datasets, and do not improve OOD generalization directly.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 CIFAR-10C neurogym cognitive tasks synthetic Gaussian point clouds
Benchmarks: CIFAR-10 CIFAR-100 linear probe CIFAR-10C
Metrics: manifold capacity test accuracy linear probe accuracy manifold dimension manifold radius center alignment axis alignment center-axis alignment
Baselines: accuracy curves weight changes NTK-label alignment representation-label alignment
Builds on: GLUE (Geometry Linked to Untangling Efficiency) manifold capacity theory Chizat lazy training inverse scale factor interpolation between lazy and rich regimes
Closest papers by meaning
- Task structure and nonlinearity jointly determine learned representational geometry (ICLR 2024 · similarity 0.89)
- Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning (NeurIPS 2024 · similarity 0.87)
- Geometry-Aware Similarity Metrics for Neural Representations on Riemannian and Statistical Manifolds (NeurIPS 2026 · similarity 0.87)
- Disentangling Representations through Multi-task Learning (ICLR 2025 · similarity 0.87)
- From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks (ICLR 2025 · similarity 0.87)
- The Importance of Being Lazy: Scaling Limits of Continual Learning (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.