Research map / Label-efficient and robust learning / Recognition and long-tail
When does dough become a bagel? Analyzing the remaining mistakes on ImageNet
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Re-label reviewed model mistakes with expert multi-label judgments | ViT-3B on ImageNet multi-label validation · ImageNet multi-label validation, All | MLA: 96.3% -> 97.9% | Helped |
| Use Greedy Soups instead of ViT-3B | ImageNet multi-label evaluation · ImageNet multi-label validation, All Classes | MLA: 97.8% -> 98.6% | Helped |
| Use Greedy Soups instead of the best human labeler | ImageNet multi-label evaluation · ImageNet multi-label validation, All Classes | MLA: 97.4% -> 98.6% | Helped |
| Use the best human labeler instead of Greedy Soup | ImageNet multi-label evaluation · ImageNet multi-label validation, Objects | MLA: 98.8% -> 99.0% | Helped |
| Remove exact duplicate validation images from training (ablation) | ResNet50 · ImageNet validation, leaked subset | Top1: 26.9% -> 40.2% | Helped |
| Remove exact duplicate validation images from training (ablation) | ViT-3B · ImageNet validation, leaked subset | MLA: 93.4% -> 94.0% | Helped |
| Remove exact duplicate validation images from training (ablation) | ViT-3B · ImageNet validation, whole set | Top1: 89.5% -> 89.4% | Hurt |
| Remove exact duplicate validation images from training (ablation) | ViT-3B · ImageNet validation, whole set | MLA: 97.8% -> 97.4% | Hurt |
| Remove exact duplicate validation images from training (ablation) | ResNet50 · ImageNet validation, whole set | Top1: 76.0% -> 76.0% | No clear effect |
| Review novel predictions with an expert panel rather than counting them as errors | ViT-3B on ImageNet multi-label validation · ImageNet multi-label validation | predictions reclassified as correct or problematic: 140 | Helped |
| Review novel predictions with an expert panel rather than counting them as errors | Greedy Soups on ImageNet · ImageNet multi-label validation | novel predictions reclassified as problematic or correct: 46.5% (92/198) | Helped |
Closest papers by meaning
- Automated Classification of Model Errors on ImageNet (NeurIPS 2023 · similarity 0.93)
- Doomed to Re-Annotate, Forever: The ImageNet Story (NeurIPS 2026 · similarity 0.92)
- Evaluating Machine Accuracy on ImageNet (ICML 2020 · similarity 0.92)
- Object Detection Benchmarks are Incomplete: The Role of Label Errors and Annotation Uncertainty (NeurIPS 2026 · similarity 0.88)
- SELECT: A Large-Scale Benchmark of Data Curation Strategies for Image Classification (NeurIPS 2024 · similarity 0.87)
- ConvNet vs Transformer, Supervised vs CLIP: Beyond ImageNet Accuracy (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.