Research map / Label-efficient and robust learning / Recognition and long-tail
On the importance of single directions for generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on datasets with randomized labels rather than true labels | MLP, ConvNet, and ResNet · MNIST, CIFAR-10, and ImageNet | Robustness to cumulative single-direction ablation | Hurt |
| Train on randomized labels rather than true labels | MLP and ConvNet · MNIST and CIFAR-10 | Robustness to random noise | Hurt |
| Use a poorly generalizing initialization, data order, or learning rate | CIFAR-10 ConvNet · CIFAR-10 | Robustness to cumulative feature-map ablation | Hurt |
| Use single-direction AUC as a training progress signal | MNIST MLP · MNIST training over time | Spearman correlation with test loss: -0.728 | Mixed |
| Use single-direction AUC to select hyperparameters | CIFAR-10 ConvNet · CIFAR-10 | Spearman correlation with test accuracy: 0.914 | Helped |
| Add dropout at the training probability (ablation) | MNIST MLP · MNIST with corrupted and unmodified labels | Robustness to cumulative ablation up to the dropout fraction | Helped |
| Rely on dropout as a regularizer beyond its training probability (ablation) | MNIST MLP · MNIST with corrupted labels | Robustness to cumulative ablation beyond the dropout fraction | No clear effect |
| Add batch normalization (ablation) | CIFAR-10 ConvNet · CIFAR-10 | Robustness to cumulative feature-map ablation | Helped |
| Add batch normalization (ablation) | CIFAR-10 ConvNet · CIFAR-10 | Feature-map class selectivity | Helped |
| Clamp ablated feature maps to their empirical mean instead of zero (ablation) | CIFAR-10 ConvNet · CIFAR-10 | Performance after cumulative ablation | Hurt |
| Use class selectivity to identify important units | MLP, ConvNet, and ResNet · MNIST, CIFAR-10, and ImageNet | Prediction of unit ablation impact: 0.095; -0.428 | No clear effect |
| Use mutual information to identify important units | MLP, ConvNet, and ResNet · MNIST, CIFAR-10, and ImageNet | Prediction of unit ablation impact | No clear effect |
Closest papers by meaning
- Sensitivity and Generalization in Neural Networks: an Empirical Study (ICLR 2018 · similarity 0.88)
- Data Distributional Properties As Inductive Bias for Systematic Generalization (CVPR 2025 · similarity 0.87)
- Do Neural Networks Need Gradient Descent to Generalize? A Theoretical Study (NeurIPS 2025 · similarity 0.87)
- Unveiling Memorization–Generalization Coexistence: A Case Study on Arithmetic Tasks with Label Noise (NeurIPS 2026 · similarity 0.87)
- To Grok or not to Grok: Disentangling Generalization and Memorization on Corrupted Algorithmic Datasets (ICLR 2024 · similarity 0.86)
- What Variables Affect Out-of-Distribution Generalization in Pretrained Models? (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.