Research map / Architectures and efficiency / RNNs and normalization
Bridging Critical Gaps in Convergent Learning: How Representational Alignment Evolves Across Layers, Training, and Distribution Shifts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Large-scale audit of representational convergence across vision models (and ViTs, MoCo, Pythia LMs, and NSD brain data) comparing linear, Procrustes, and permutation/soft-matching alignment across layers, training checkpoints, and OOD shifts.
Problem
Prior convergent-learning studies examine few models and one dataset, use a single alignment metric, and evaluate only final checkpoints; it is unclear which minimal transformations align representations, when alignment emerges during training, and how it behaves under distribution shift.
Approach
Compare three alignment families with progressively stricter invariances (linear regression, orthogonal Procrustes, permutation/soft-matching) across layers, early training epochs, and 17 OOD image datasets, using center-pixel channel activations to make comparisons tractable.
Key results
- Procrustes alignment is nearly as high as linear regression across ResNet/VGG layers, while permutation scores are lower but substantially above chance.
- Most alignment emerges within the first training epoch, well before task accuracy plateaus; early conv layers show little training-induced change.
- Under 17 OOD shifts, early-layer alignment stays stable while deeper-layer alignment diverges and correlates with OOD accuracy; random rotation reduces permutation alignment by up to ~202% for ResNet18 layers.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use orthogonal Procrustes alignment instead of unrestricted linear alignment | Vision models, CNN layer representations · ImageNet | representation alignment | No clear effect |
| Restrict alignment to permutation matching rather than Procrustes or linear mapping | Vision models, CNN layer representations · ImageNet | representation alignment | Hurt |
| Train independently initialized networks rather than use randomly initialized networks | CNNs · ImageNet | Procrustes alignment: +145.26% in early layers; +493.84% in deeper layers | Helped |
| Apply a random rotation to the unit basis (ablation) | ResNet18 · ImageNet, convolutional layers | permutation alignment: 10.47% / 201.84% | Hurt |
| Apply a random rotation to the unit basis (ablation) | ResNet50 · ImageNet, convolutional layers | permutation alignment: 2.61% / 180.75% | Hurt |
| Apply a random rotation to the unit basis (ablation) | VGG16 · ImageNet, convolutional layers | permutation alignment: 21.72% / 81.12% | Hurt |
| Apply a random rotation to the unit basis (ablation) | VGG19 · ImageNet, convolutional layers | permutation alignment: 41.95% / 89.68% | Hurt |
| Apply a random rotation to the unit basis (ablation) | CNNs · CIFAR100, convolutional layers | permutation alignment: ResNet18: 6.40% / 51.26%; ResNet50: -3.38% / 35.29%; VGG16: 5.66% / 63.97%; VGG19: 2.15% / 35.36% | Mixed |
| Use a random activation pixel instead of the center pixel (ablation) | CNNs · ImageNet | representation alignment | No clear effect |
| Use Spearman rank correlation instead of Pearson correlation to report alignment (ablation) | Vision models · ImageNet | representation alignment | No clear effect |
| Evaluate on out-of-distribution images instead of ImageNet test images | ImageNet-trained CNNs · 17 OOD datasets | Procrustes alignment | Mixed |
| Continue training beyond the first epoch | CNNs · ImageNet and CIFAR100 | Procrustes alignment | No clear effect |
Surprising findings
- Alignment emerges within the first epoch, suggesting convergence is driven by input statistics and architecture rather than the final task solution.
- ViTs show no privileged axes: permutation scores are unchanged after random rotation, unlike CNNs.
Limitations
- Center-pixel approximation enforces strict spatial correspondence and may underestimate alignment.
- Cannot resolve when within the first epoch convergence occurs; alignment metrics may hide task-relevant changes.
- Theoretical account of rapid early convergence is not provided.
Experimental setup
Datasets: CIFAR100 ImageNet Pile STSB Natural Scenes Dataset ImageNet-C-style OOD suite (Stylized ImageNet, Sketch, Edges, etc.)
Benchmarks: ImageNet CIFAR100 17 OOD datasets (Stylized ImageNet and related) STSB Natural Scenes Dataset
Metrics: linear regression alignment orthogonal Procrustes alignment permutation/soft-matching alignment Spearman correlation top-1 accuracy
Baselines: linear regression alignment permutation alignment SVCCA randomly initialized networks
Builds on: canonical correlation analysis centered kernel alignment Platonic Representation Hypothesis contravariance principle MoCo
Closest papers by meaning
- Not all solutions are created equal: An analytical dissociation of functional and representational similarity in deep linear neural networks (ICML 2025 · similarity 0.89)
- Sparse components distinguish visual pathways & their alignment to neural networks (ICLR 2025 · similarity 0.89)
- Position: The Platonic Representation Hypothesis (ICML 2024 · similarity 0.88)
- Model–Behavior Alignment under Flexible Evaluation: When the Best-Fitting Model Isn’t the Right One (NeurIPS 2025 · similarity 0.88)
- Evaluating alignment between humans and neural network representations in image-based learning tasks (NeurIPS 2024 · similarity 0.88)
- Formation of Representations in Neural Networks (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.