Research map / Label-efficient and robust learning / Recognition and long-tail
Do ImageNet Classifiers Generalize to ImageNet?
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, Vaishaal Shankar
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate existing classifiers on newly collected CIFAR-10 images | CIFAR-10 classifiers · CIFAR-10.1 | accuracy: 98.4 -> 95.5 | Hurt |
| Evaluate existing classifiers on newly collected CIFAR-10 images | CIFAR-10 classifiers · CIFAR-10.1 | accuracy: 97.1 -> 93.0 | Hurt |
| Evaluate existing classifiers on newly collected ImageNet images | ImageNet classifiers · ImageNetV2 MatchedFrequency | top-1 accuracy: 82.9 -> 72.2 | Hurt |
| Evaluate existing classifiers on newly collected ImageNet images | ImageNet classifiers · ImageNetV2 MatchedFrequency | top-1 accuracy: 82.5 -> 72.2 | Hurt |
| Increase the average MTurk selection frequency of sampled images using TopImages instead of MatchedFrequency | ImageNet classifiers · ImageNetV2 | average top-1 accuracy change: -11.8% -> +2.1% | Improved |
| Use Threshold0.7 sampling rather than MatchedFrequency | ImageNet classifiers · ImageNetV2 | average top-1 accuracy change: -11.8% -> -3.2% | Improved |
| Increase original-test accuracy across models | CIFAR-10 classifiers · CIFAR-10.1 | accuracy: slope 1.69 | Improved |
| Increase original-test accuracy across models | ImageNet classifiers · ImageNetV2 MatchedFrequency | top-1 accuracy: slope 1.11 | Improved |
| Retune learning rate, dropout, and weight decay for VGG | vgg16_keras · CIFAR-10.1 | accuracy: 85.3% -> 85.8% | No clear effect |
| Add about 1,000 CIFAR-10.1 images to the training set | vgg16_keras · Held-out CIFAR-10.1 | accuracy: 84.9% -> 85.1% and 85.4% | No clear effect |
| Evaluate human accuracy on the hardest images from the new rather than original CIFAR-10 test set | Human participants · Hardest CIFAR-10 images | average accuracy difference: -0.8% | No clear effect |
| Train an ℓ∞-robust CIFAR-10 model and evaluate it on the new test set | ℓ∞-robust CIFAR-10 model · CIFAR-10.1 | accuracy | No clear effect |
Closest papers by meaning
- CIFAR-10-Warehouse: Broad and More Realistic Testbeds in Model Generalization Analysis (ICLR 2024 · similarity 0.87)
- Can Biases in ImageNet Models Explain Generalization? (CVPR 2024 · similarity 0.86)
- A Decade's Battle on Dataset Bias: Are We There Yet? (ICLR 2025 · similarity 0.86)
- Doomed to Re-Annotate, Forever: The ImageNet Story (NeurIPS 2026 · similarity 0.86)
- Improving Generalization via Meta-Learning on Hard Samples (CVPR 2024 · similarity 0.85)
- Rethinking the Evaluation Protocol of Domain Generalization (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
