Research map / Architectures and efficiency / RNNs and normalization
Diffused Redundancy in Pre-trained Representations
Vedant Nanda, Till Speicher, John P Dickerson, Krishna P. Gummadi, Soheil Feizi, Adrian Weller
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| adversarial training instead of standard training | ResNet50 · CIFAR10, CIFAR100, Flowers, Pets | diffused redundancy | Improved |
| shrink penultimate layer width to 8 neurons (ResNet50_ff8) (ablation) | ResNet50 · CIFAR10, CIFAR100, Flowers, Pets | diffused redundancy | Hurt |
| more downstream classes (Flowers, CIFAR100 vs CIFAR10) | ResNet50 · downstream transfer | neurons needed: 0.20 -> 0.90 | Hurt |
| random subset of neurons vs top-k principal components, k > 20% | ResNet50 · downstream linear probe | accuracy | No clear effect |
| random subset vs top-k PCA for small k | ResNet50 · downstream linear probe | accuracy | Hurt |
| random neurons vs bottom-k principal components | ResNet50 · downstream linear probe | accuracy | Improved |
| apply dropout (0.1 to 0.8) or DeCov regularizer during pretraining (ablation) | ResNet50 · downstream transfer | diffused redundancy | No clear effect |
| pretrain on larger upstream dataset (ImageNet21k) | ResNet50 · downstream transfer | diffused redundancy | Improved |
| MRL loss evaluated on random neurons vs first-k optimized neurons | ResNet50 MRL · downstream transfer | accuracy | Mixed |
| drop neurons randomly | ResNet50 · class-wise accuracy | Gini coefficient | Hurt |
| MRL loss vs other losses while dropping neurons | ResNet50 · class-wise accuracy | Gini coefficient | Hurt |
| drop neurons randomly on harder tasks (ImageNet1k, Places365) | ResNet50 · ImageNet1k, Places365 | accuracy | No clear effect |
Closest papers by meaning
- Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks (ICML 2024 · similarity 0.87)
- Exploring the Limits of Large Scale Pre-training (ICLR 2022 · similarity 0.87)
- Rethinking Supervised Pre-Training for Better Downstream Transferring (ICLR 2022 · similarity 0.87)
- How Not to Stitch Representations to Measure Similarity: Task Loss Matching Versus Direct Matching (AAAI 2025 · similarity 0.86)
- Tracing Representation Progression: Analyzing and Enhancing Layer-Wise Similarity (ICLR 2025 · similarity 0.86)
- On the Emergence of Cross-Task Linearity in Pretraining-Finetuning Paradigm (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
