Research map / Label-efficient and robust learning / Recognition and long-tail

Doomed to Re-Annotate, Forever: The ImageNet Story

NeurIPS 2026 · Label-efficient and robust learning › Recognition and long-tail · approach: Loss design and classifier theory · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Re-annotates the full ImageNet-1k validation set from scratch with multilabel bounding boxes, revised class definitions, and attributes (ReImageNet), then re-evaluates MLLMs, VLMs and supervised models on the corrected labels.

Problem

ImageNet-1k validation labels are noisy, single-label, and taxonomically problematic, yet top-1 accuracy on them remains the dominant metric; prior reannotations kept the flawed taxonomy and were inconsistent with each other.

Approach

A from-scratch, iterative reannotation by trained in-house annotators working in WordNet-derived class groups, with multilabel bounding boxes, revised class names/definitions, and five visual attributes, assisted by OWLv2/OpenCLIP and later MLLM+SAM3 proposals; evaluated with multilabel ReaL-style accuracy plus crop-based single-label protocols.

Key results

• About 12% of original ImageNet-1k validation labels are incorrect; 33.3% of images are multilabel and 3.8% contain no ImageNet-1k object.
• Top-1 accuracy on corrected labels rises by 5.1-6.1% for MLLMs (GPT-5.4, Qwen3-VL, Gemma 4) but only up to 1.2% for supervised models.
• Supervised models lose up to 33.5% accuracy on object crops (EVA-02), versus smaller drops for MLLMs.

Surprising findings

• Accuracy stays broadly stable as the number of labels per image grows, contrary to the expectation of a more target-rich environment.
• Supervised models show a much larger drop on rendition images than MLLMs, and EVA-02 gains almost nothing from corrected labels (+0.05%).

Limitations

• Annotators share a European background and are non-experts in fine-grained domains; model predictions may nudge labelling.
• Relies on closed-source MLLMs (GPT-4o, GPT-5.4) whose outputs are not repeatable.
• Covers only the validation set; images date from 2012.

Experimental setup

Datasets: ImageNet-1k validation set

Benchmarks: ImageNet-1k ImageNet-V2 ImageNet-A ImageNet-R ImageNet-Sketch ObjectNet CounterAnimal

Metrics: top-1 accuracy ReaL accuracy label error rate lower bound

Baselines: ImageNet ReaL ImageNet-X ImageNetMultiLabel Label Errors

Builds on: ImageNet ReaL OWLv2 SAM3 WordNet

Closest papers by meaning

• When does dough become a bagel? Analyzing the remaining mistakes on ImageNet (NeurIPS 2022 · similarity 0.92)
• Re-Labeling ImageNet: From Single to Multi-Labels, From Global to Localized Labels (CVPR 2021 · similarity 0.92)
• Automated Classification of Model Errors on ImageNet (NeurIPS 2023 · similarity 0.91)
• Evaluating Machine Accuracy on ImageNet (ICML 2020 · similarity 0.90)
• Object Detection Benchmarks are Incomplete: The Role of Label Errors and Annotation Uncertainty (NeurIPS 2026 · similarity 0.88)
• SELECT: A Large-Scale Benchmark of Data Curation Strategies for Image Classification (NeurIPS 2024 · similarity 0.87)