Research map / Label-efficient and robust learning / Noisy labels
dopanim: A Dataset of Doppelganger Animals with Noisy Annotations from Multiple Humans
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces dopanim, a dataset of about 15,750 doppelganger animal images in 15 classes annotated with soft likelihoods and annotator metadata by 20 humans, and benchmarks multi-annotator learning approaches on seven annotation subsets.
Problem
Learning from noisy human annotations needs realistic datasets, but existing ones are mostly easy tasks with hard labels, few annotations per annotator, and no annotator metadata.
Approach
A challenging fine-grained animal classification dataset (doppelganger species groups) with human-estimated unnormalized likelihood labels, per-annotator metadata from questionnaires, and annotation times, plus seven controlled annotation-subset variants for benchmarking multi-annotator learning.
Key results
- annot-mix achieved the best accuracy on most dopanim variants, e.g., 84.2% on full and 35.2% on worst-1 versus 27.0% for majority-vote training on worst-1.
- Ground-truth training reaches 89.0% test accuracy, showing a large gap to noisy-label training.
- Human-estimated likelihoods as weights in majority voting improved accuracy by +9.0 points on worst-1 and +0.8 points on full.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with multi-annotator learning instead of majority-vote labels | dopanim benchmark approaches · dopanim worst variants | test accuracy | Helped |
| use human-estimated likelihoods as majority-voting weights | mv-base · dopanim variants | test accuracy: +9.0_{± 0.8}; +8.4_{± 0.5}; +5.3_{± 0.5}; -0.7_{± 0.6}; +2.2_{± 0.7}; +1.5_{± 0.3}; +0.8_{± 0.3} | Mixed |
| add annotator metadata | annot-mix · dopanim variants | test accuracy: +1.1_{± 1.9}; -0.2_{± 0.8}; +0.4_{± 0.8}; +0.0_{± 0.3}; +0.3_{± 0.5}; +0.3_{± 0.4}; +0.3_{± 0.4} | Mixed |
| use uncertainty sampling instead of random sampling | DINOv2 ViT-S/14 active learning · dopanim active learning | accuracy | Helped |
| select instances by uncertainty sampling instead of random sampling | DINOv2 ViT-S/14 active learning · dopanim active learning | annotation time: almost 1 s longer | Hurt |
Surprising findings
- Instance-dependent annotator performance models are not consistently better than class-dependent ones.
- Majority-vote training gives competitive Brier scores and calibration errors on some variants despite inferior accuracy.
Limitations
- Dataset is small-scale due to high annotation costs.
- Benchmark limited to seven dopanim variants and the DINOv2 ViT-S/14 backbone.
- Annotators are 20 student assistants from one institution.
Experimental setup
Datasets: dopanim iNaturalist
Benchmarks: dopanim
Metrics: test accuracy Brier score top-label calibration error
Baselines: cl trace-reg conal union-net madl geo-reg-w geo-reg-f crowd-ar annot-mix majority voting
Builds on: DINOv2 mixup annot-mix geo-reg
Closest papers by meaning
- Doppelgangers and Adversarial Vulnerability (CVPR 2025 · similarity 0.83)
- CleverBirds: A Multiple-Choice Benchmark for Fine-grained Human Knowledge Tracing (NeurIPS 2025 · similarity 0.83)
- Noisy Ostracods: A Fine-Grained, Imbalanced Real-World Dataset for Benchmarking Robust Machine Learning and Label Correction Methods (NeurIPS 2024 · similarity 0.83)
- Animal-Bench: Benchmarking Multimodal Video Models for Animal-centric Video Understanding (NeurIPS 2024 · similarity 0.83)
- Fish-Vista: A Multi-Purpose Dataset for Understanding & Identification of Traits from Images (CVPR 2025 · similarity 0.82)
- Quantifying Uncertainty in Error Consistency: Towards Reliable Behavioral Comparison of Classifiers (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.