Research map / Label-efficient and robust learning / Recognition and long-tail
Realistic Evaluation of Deep Partial-Label Learning Algorithms
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PLENCH, a benchmark for deep partial-label learning that studies model selection criteria (CR, AA, OA) and introduces PLCIFAR10, a human-annotated partial-label CIFAR-10 dataset, to systematically compare 27 PLL algorithms.
Problem
Deep partial-label learning evaluation is inconsistent: model selection is rarely studied and often relies on clean validation labels, experimental settings vary across papers, and real-world image datasets with human-annotated partial labels are lacking.
Approach
Proposes partial-label-only model selection criteria (Covering Rate, Approximated Accuracy) with theoretical consistency results, a unified benchmark of 27 algorithms over 11 datasets, and a new crowdsourced image dataset PLCIFAR10 collected via MTurk.
Key results
- On PLCIFAR10-Aggregate with ResNet, ABLE reaches 86.30 accuracy with AA selection and PRODEN 85.95 with CR, while CR is competitive with OA-with-early-stopping in most cases.
- On PLCIFAR10-Vaguest with ResNet, DIRK obtains the best results across all four criteria (e.g., 80.32 with CR, 81.08 with OA and ES).
- The paper reports that many early vanilla algorithms, such as PRODEN and Forward, remain competitive with newer complicated methods, and no single algorithm wins in all cases.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CR for model selection instead of AA | PLL algorithms on the PLCIFAR10 image datasets · PLCIFAR10 | accuracy | Helped |
| Use AA for model selection | PLL algorithms when model outputs are unreliable · PLCIFAR10 | accuracy | Hurt |
| Introduce noisy partial labels | most PLL algorithms · PLCIFAR10-Vaguest | accuracy | Hurt |
| Use PRODEN | deep PLL algorithms · Real tabular datasets | accuracy | Helped |
| Use AA for model selection when classifiers are not accurate enough | PLL algorithms · Real tabular datasets | accuracy | Hurt |
Surprising findings
- Many concise early PLL algorithms, such as PRODEN, match or outperform later complicated holistic methods.
- AA model selection can fail badly when the classifier is inaccurate, e.g., MCL-MSE drops to 10% on PLCIFAR10-Aggregate with AA.
Limitations
- Datasets are relatively small (largest is 50,000 images), and large-scale PLL datasets are left to future work.
- Experiments use simple deep networks (ResNet, DenseNet, MLP) rather than foundation models.
Experimental setup
Datasets: PLCIFAR10 CIFAR-10 Lost Soccer Player Yahoo! News MSRCv2 Mirflickr Birdsong Malagasy Italian English
Benchmarks: PLCIFAR10-Aggregate PLCIFAR10-Vaguest Lost Soccer Player Yahoo! News MSRCv2 Mirflickr Birdsong Malagasy Italian English
Metrics: classification accuracy covering rate approximated accuracy oracle accuracy
Baselines: PRODEN CAVL POP ABS-MAE ABS-GCE EXP MCL-GCE MCL-MSE CC LWS IDGP PC
Builds on: PRODEN CAVL PiCO FREDIS
Closest papers by meaning
- What Makes Partial-Label Learning Algorithms Effective? (NeurIPS 2024 · similarity 0.91)
- Conformal Prediction for Partial Label Learning (AAAI 2025 · similarity 0.91)
- CroSel: Cross Selection of Confident Pseudo Labels for Partial-Label Learning (CVPR 2024 · similarity 0.90)
- Partial Label Learning with a Partner (AAAI 2024 · similarity 0.90)
- Does Label Smoothing Help Deep Partial Label Learning? (ICML 2024 · similarity 0.89)
- Distilling Reliable Knowledge for Instance-Dependent Partial Label Learning (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.