Research map / Architectures and efficiency / Pruning and efficient tuning
What Do Neural Networks Learn When Trained With Random Labels?
Hartmut Maennel, Ibrahim M Alabdulmohsin, Ilya O Tolstikhin, Robert Baldock, Olivier Bousquet, Sylvain Gelly, Daniel Keysers
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add sampled convolutional layer 1 to the initialization (ablation) | Simple CNN · CIFAR10, 100 iterations | train accuracy: 0.31 -> 0.34 | Improved |
| Add sampled convolutional layer 2 (ablation) | Simple CNN with layer 1 sampled · CIFAR10, 100 iterations | train accuracy: 0.34 -> 0.38 | Improved |
| Add sampled convolutional layer 3 (ablation) | Simple CNN with layers 1–2 sampled · CIFAR10, 100 iterations | train accuracy: 0.38 -> 0.41 | Improved |
| Add sampled convolutional layers 1–3 rather than sampling none (ablation) | Simple CNN · CIFAR10, 100 iterations | test accuracy: 0.31 -> 0.40 | Improved |
| Add sampled convolutional layer 1 to the initialization (ablation) | Simple CNN · CIFAR10, 1000 iterations | train accuracy: 0.58 -> 0.61 | Improved |
| Add sampled convolutional layer 2 (ablation) | Simple CNN with layer 1 sampled · CIFAR10, 1000 iterations | train accuracy: 0.61 -> 0.67 | Improved |
| Add sampled convolutional layer 3 (ablation) | Simple CNN with layers 1–2 sampled · CIFAR10, 1000 iterations | train accuracy: 0.67 -> 0.68 | Improved |
| Add sampled convolutional layer 1 to the initialization (ablation) | Simple CNN · CIFAR10, 1000 iterations | test accuracy: 0.53 -> 0.55 | Improved |
| Add sampled convolutional layer 2 (ablation) | Simple CNN with layer 1 sampled · CIFAR10, 1000 iterations | test accuracy: 0.55 -> 0.56 | Improved |
| Add sampled convolutional layer 3 (ablation) | Simple CNN with layers 1–2 sampled · CIFAR10, 1000 iterations | test accuracy: 0.56 -> 0.56 | No clear effect |
| Pre-train on random labels before downstream training, while matching weight norms to initialization | VGG16 and ResNet18 · CIFAR10 and ImageNet downstream tasks | training speed | Improved |
| Initialize first-layer filters from the covariance learned during random-label training instead of using the pre-trained filters | Simple CNN · CIFAR10 downstream tasks | training speed | Improved |
Closest papers by meaning
- Learning with Structural Labels for Learning with Noisy Labels (CVPR 2024 · similarity 0.87)
- Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels (ICML 2019 · similarity 0.87)
- The Early Phase of Neural Network Training (ICLR 2020 · similarity 0.87)
- Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks (ICLR 2024 · similarity 0.85)
- Pretraining with Random Noise for Fast and Robust Learning without Weight Transport (NeurIPS 2024 · similarity 0.85)
- Directional Label Diffusion Model for Learning from Noisy Labels (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
