Research map / Architectures and efficiency / RNNs and normalization
Insights on representational similarity in neural networks with canonical correlation
Ari Morcos, Maithra Raghu, Samy Bengio
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use projection weighting instead of an unweighted mean of canonical correlations (ablation) | CCA representation similarity · Synthetic signal-and-noise representations | robustness to signal-to-noise ratio | Improved |
| Train on true labels rather than a fixed random label permutation | CIFAR-10 CNNs · CIFAR-10, later layers | pairwise PWCCA distance | Improved |
| Evaluate softmax representations on test data rather than training data | CIFAR-10 CNNs trained on true or random labels · CIFAR-10 test data, softmax | pairwise PWCCA distance | Mixed |
| Increase network width | CIFAR-10 CNNs · CIFAR-10, trained networks across widths | pairwise CCA distance: approximately 0.47 to 0.28 | Improved |
| Increase network width | CIFAR-10 CNNs · CIFAR-10, networks at initialization across widths | pairwise CCA distance: approximately 0.67 to 0.63 | No clear effect |
| Train otherwise identical networks with different learning rates | CIFAR-10 CNNs · CIFAR-10, eighth layer | pairwise CCA distance | Mixed |
| Use projection weighting rather than unweighted CCA (ablation) | RNN representations during training · Penn Treebank and WikiText-2 | clarity of bottom-up convergence | Improved |
| Compare representations with cosine or Euclidean distance instead of CCA (ablation) | RNN representations during training · Penn Treebank and WikiText-2 | clarity of bottom-up convergence | Hurt |
| Use unweighted CCA or SVCCA instead of projection-weighted CCA (ablation) | CNN representations across generalizing and memorizing networks · CIFAR-10, layer-wise pairwise distance | representation similarity fidelity | Hurt |
| Compare hidden states across sequence steps with CCA instead of cosine or Euclidean distance (ablation) | Toy RNNs with rotated hidden states · Toy RNN sequence timesteps | detection of linear representational similarity | Improved |
| Present repeated inputs after many unique sequence inputs rather than early in the sequence | LSTM hidden states · Repeated-input sequence timesteps | CCA similarity across timesteps | Hurt |
| Use distinct sequence inputs rather than repeated inputs | LSTM hidden states · PTB and WikiText-2 sequence timesteps | linear similarity across timesteps | Hurt |
Closest papers by meaning
- Tracing Representation Progression: Analyzing and Enhancing Layer-Wise Similarity (ICLR 2025 · similarity 0.89)
- Spectral Analysis of Representational Similarity with Limited Neurons (NeurIPS 2025 · similarity 0.89)
- Geometry-Aware Similarity Metrics for Neural Representations on Riemannian and Statistical Manifolds (NeurIPS 2026 · similarity 0.88)
- Formation of Representations in Neural Networks (ICLR 2025 · similarity 0.87)
- When Representations Align: Universality in Representation Learning Dynamics (ICML 2024 · similarity 0.87)
- Not all solutions are created equal: An analytical dissociation of functional and representational similarity in deep linear neural networks (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
