Research map / Label-efficient and robust learning / Noisy labels
Noisy Ostracods: A Fine-Grained, Imbalanced Real-World Dataset for Benchmarking Robust Machine Learning and Label Correction Methods
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Noisy Ostracods dataset (71,466 expert-annotated ostracod images with open-set, pseudo-class, mixed-class and feature noise) and benchmarks robust learning and label correction methods, finding that CE training matches or beats most LNL methods and NECV ensemble cross-validation is best for label correction.
Problem
Real-world fine-grained, highly imbalanced taxonomy datasets contain diverse label and feature noise that existing learning-with-noisy-labels and label correction methods have not been evaluated on.
Approach
A real-world noisy fine-grained ostracod dataset with categorized noise sources (feature errors, pseudo/mixed/new classes, hard classes), manually cleaned test and validation sets, and a simple naive ensemble cross-validation baseline (NECV) compared against LNL and label-correction methods.
Key results
- Standard cross-entropy with ImageNet pretraining (ResNet-50) reached 95.98% accuracy, and none of the LNL methods gave substantial gains over it.
- NECV ensemble cross-validation achieved a 71.19% label-error hit rate, versus 59.37% for Confident Learning and 26-31% for SimiFeat variants.
- The transition-matrix method with the ground-truth matrix reached only 84.22% accuracy (ResNet-50), trained on 51 classes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PLM instead of CE training | ResNet-50 on Noisy Ostracods · Noisy Ostracods test/validation | accuracy: 95.98 -> 96.77 | Helped |
| Use PLM instead of CE training | ViT-B-16 on Noisy Ostracods · Noisy Ostracods test/validation | accuracy: 95.03 -> 96.36 | Helped |
| Use robust-learning methods instead of standard CE training | ImageNet-pretrained models on Noisy Ostracods · Noisy Ostracods test/validation | F1: 79.51 (CE) vs 78.21 (Co-teaching+) | Hurt |
| Train with the true class-dependent transition matrix on 51 classes | Noisy Ostracods · Noisy Ostracods test/validation | accuracy: 95.98 (CE) vs 84.22 (Transition, ResNet-50); 95.03 (CE) vs 83.28 (Transition, ViT-B-16) | Hurt |
| Use NECV instead of other label-correction methods | Label error detection on Noisy Ostracods · Noisy Ostracods test/validation | Hit rate: 71.19 (NECV) vs 59.37 (CL); 53.71 (CINCER) | Helped |
| Fine-tune the MAE feature extractor for SimiFeat (ablation) | SimiFeat label correction · Noisy Ostracods test/validation | Hit rate: 27.55 -> 28.18 | Helped |
| Use majority voting with fine-tuned MAE features (ablation) | SimiFeat label correction · Noisy Ostracods test/validation | Hit rate: 28.18 -> 31.07 | Helped |
| Use fine-tuned MAE features with majority voting (ablation) | SimiFeat label correction · Noisy Ostracods, Paradoxostomatid class | Hit rate: 14.29 (MAE-raw-mv) vs 2.65 (MAE-trained-mv) | Hurt |
| Use AUM instead of NECV | Label error detection on Noisy Ostracods · Noisy Ostracods test/validation | Hit rate: 71.19 -> 29.56 | Hurt |
| Use NECV while excluding Paradoxostomatid | Label error detection on Noisy Ostracods · Noisy Ostracods test/validation, without Paradoxostomatid | Hit rate: 71.19 -> 91.58 | Helped |
| Use SimiFeat-MAE instead of CLIP features (ablation) | SimiFeat label correction · Noisy Ostracods test/validation | Hit rate: 26.29 (SimiFeat-CLIP) vs 28.18 (MAE-trained); 27.55 (MAE-raw) | Mixed |
| Use majority voting instead of HOC ranking with SimiFeat-CLIP (ablation) | SimiFeat-CLIP · Noisy Ostracods test/validation | Hit rate: 26.29 -> 26.29 | No clear effect |
Surprising findings
- Standard cross-entropy training achieved a lower error rate (4.02%) than the dataset's estimated noise rate (5.58%), and outperformed most dedicated LNL methods.
- Even with the ground-truth noise transition matrix, the class-dependent noisy-posterior method performed poorly, suggesting class-dependent noise assumptions do not hold.
Limitations
- No fully cleaned training set is provided; cleaned validation and test sets only cover genus level.
- Taxonomy is not static, so annotations may change with future revisions.
- Transition-matrix experiments use ground-truth matrices (information leakage) and only 51 classes, so results are not directly comparable.
Experimental setup
Datasets: Noisy Ostracods ImageNet Endless Forams
Benchmarks: Noisy Ostracods
Metrics: accuracy precision recall F1 hit rate fix precision
Baselines: CE Mixup+Cutmix Confident Learning Loss-clip Co-teaching Co-teaching+ DivideMix SAM PLM SDN Label Wave SimiFeat
Builds on: Co-teaching DivideMix Confident Learning SimiFeat AUM
Closest papers by meaning
- BIOSCAN-5M: A Multimodal Dataset for Insect Biodiversity (NeurIPS 2024 · similarity 0.84)
- dopanim: A Dataset of Doppelganger Animals with Noisy Annotations from Multiple Humans (NeurIPS 2024 · similarity 0.83)
- DivShift: Exploring Domain-Specific Distribution Shifts in Large-Scale, Volunteer-Collected Biodiversity Datasets (AAAI 2025 · similarity 0.82)
- Open-Insect: Benchmarking Open-Set Recognition of Novel Species in Biodiversity Monitoring (NeurIPS 2025 · similarity 0.82)
- Fish-Vista: A Multi-Purpose Dataset for Understanding & Identification of Traits from Images (CVPR 2025 · similarity 0.82)
- The iNaturalist Sounds Dataset (NeurIPS 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.