Research map / Label-efficient and robust learning / Recognition and long-tail
Evaluating Machine Accuracy on ImageNet
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use multi-label annotations instead of top-1 accuracy for evaluation | ImageNet models (72 models) · ImageNet val | accuracy (median change, multi-label vs top-1): median improvement of 8.9% | Helped |
| Compare multi-label accuracy to top-5 accuracy | ImageNet models (72 models) · ImageNet val | accuracy (median change, multi-label vs top-5): median drop of 7.4% | No clear effect |
| Evaluate on distribution shift from ImageNet val to ImageNetV2 (model vs human robustness) | All models vs trained human labelers · ImageNet val vs ImageNetV2 multi-label | accuracy gap: models drop 5% to 8%; humans within 1% | Hurt |
| Restrict evaluation to object classes (590) instead of all 1000 classes | Human labelers vs FixResNeXt · ImageNet val objects only | multi-label accuracy: Best human 99.1-99.8% vs best model 95.0%/89.1% | Helped |
| Restrict to images humans labeled quickly (median time <= 60 s) | Human labelers and models · Fast images subset | multi-label accuracy: accuracies rise for both humans and models (e.g., FixResNeXt 96.2 / 92.3) | Helped |
| Exclude dog classes (118) from evaluation | Best model vs human labelers · ImageNet val without dogs | multi-label accuracy: best model +0.6 (ImageNet), humans mean +1.9 | Helped |
Closest papers by meaning
- When does dough become a bagel? Analyzing the remaining mistakes on ImageNet (NeurIPS 2022 · similarity 0.92)
- Doomed to Re-Annotate, Forever: The ImageNet Story (NeurIPS 2026 · similarity 0.90)
- Automated Classification of Model Errors on ImageNet (NeurIPS 2023 · similarity 0.89)
- Enriching ImageNet With Human Similarity Judgments and Psychological Embeddings (CVPR 2021 · similarity 0.89)
- Human and AI Perceptual Differences in Image Classification Errors (AAAI 2025 · similarity 0.86)
- Object Detection Benchmarks are Incomplete: The Role of Label Errors and Annotation Uncertainty (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.