AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

Diffused Redundancy in Pre-trained Representations

Vedant Nanda, Till Speicher, John P Dickerson, Krishna P. Gummadi, Soheil Feizi, Adrian Weller

NeurIPS 2023 · Architectures and efficiency › RNNs and normalization · approach: Representation geometry and collapse · 2 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
adversarial training instead of standard trainingResNet50 · CIFAR10, CIFAR100, Flowers, Petsdiffused redundancyImproved
shrink penultimate layer width to 8 neurons (ResNet50_ff8) (ablation)ResNet50 · CIFAR10, CIFAR100, Flowers, Petsdiffused redundancyHurt
more downstream classes (Flowers, CIFAR100 vs CIFAR10)ResNet50 · downstream transferneurons needed: 0.20 -> 0.90Hurt
random subset of neurons vs top-k principal components, k > 20%ResNet50 · downstream linear probeaccuracyNo clear effect
random subset vs top-k PCA for small kResNet50 · downstream linear probeaccuracyHurt
random neurons vs bottom-k principal componentsResNet50 · downstream linear probeaccuracyImproved
apply dropout (0.1 to 0.8) or DeCov regularizer during pretraining (ablation)ResNet50 · downstream transferdiffused redundancyNo clear effect
pretrain on larger upstream dataset (ImageNet21k)ResNet50 · downstream transferdiffused redundancyImproved
MRL loss evaluated on random neurons vs first-k optimized neuronsResNet50 MRL · downstream transferaccuracyMixed
drop neurons randomlyResNet50 · class-wise accuracyGini coefficientHurt
MRL loss vs other losses while dropping neuronsResNet50 · class-wise accuracyGini coefficientHurt
drop neurons randomly on harder tasks (ImageNet1k, Places365)ResNet50 · ImageNet1k, Places365accuracyNo clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy