AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Insights on representational similarity in neural networks with canonical correlation

Ari Morcos, Maithra Raghu, Samy Bengio

NeurIPS 2018 · Architectures and efficiency › RNNs and normalization · approach: Mechanistic circuit interpretability · 57 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use projection weighting instead of an unweighted mean of canonical correlations (ablation)CCA representation similarity · Synthetic signal-and-noise representationsrobustness to signal-to-noise ratioImproved
Train on true labels rather than a fixed random label permutationCIFAR-10 CNNs · CIFAR-10, later layerspairwise PWCCA distanceImproved
Evaluate softmax representations on test data rather than training dataCIFAR-10 CNNs trained on true or random labels · CIFAR-10 test data, softmaxpairwise PWCCA distanceMixed
Increase network widthCIFAR-10 CNNs · CIFAR-10, trained networks across widthspairwise CCA distance: approximately 0.47 to 0.28Improved
Increase network widthCIFAR-10 CNNs · CIFAR-10, networks at initialization across widthspairwise CCA distance: approximately 0.67 to 0.63No clear effect
Train otherwise identical networks with different learning ratesCIFAR-10 CNNs · CIFAR-10, eighth layerpairwise CCA distanceMixed
Use projection weighting rather than unweighted CCA (ablation)RNN representations during training · Penn Treebank and WikiText-2clarity of bottom-up convergenceImproved
Compare representations with cosine or Euclidean distance instead of CCA (ablation)RNN representations during training · Penn Treebank and WikiText-2clarity of bottom-up convergenceHurt
Use unweighted CCA or SVCCA instead of projection-weighted CCA (ablation)CNN representations across generalizing and memorizing networks · CIFAR-10, layer-wise pairwise distancerepresentation similarity fidelityHurt
Compare hidden states across sequence steps with CCA instead of cosine or Euclidean distance (ablation)Toy RNNs with rotated hidden states · Toy RNN sequence timestepsdetection of linear representational similarityImproved
Present repeated inputs after many unique sequence inputs rather than early in the sequenceLSTM hidden states · Repeated-input sequence timestepsCCA similarity across timestepsHurt
Use distinct sequence inputs rather than repeated inputsLSTM hidden states · PTB and WikiText-2 sequence timestepslinear similarity across timestepsHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy