Research map / Label-efficient and robust learning / Noisy labels
Unbiased Multi-Label Learning from Crowdsourced Annotations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLEAR, an unbiased risk estimator and decoupled autoencoder framework for training multi-label classifiers directly from crowdsourced multi-annotator labels, with a generalization bound.
Problem
Crowdsourced multi-label learning: training a multi-label predictor that generalizes to unseen instances when only noisy labels from multiple annotators are available for training, rather than requiring crowdsourced labels at test time.
Approach
Derives an unbiased risk estimator for CMLL from class-dependent instance-independent transition matrices, aggregates annotators into one noisy label vector with aggregated transition matrices estimated via anchor points, and couples two VAEs (feature and label) with latent distillation to capture label correlations.
Key results
- On Mirflickr with transition setting (0.2, 0.5), CLEAR improves the best baseline by 28.64% Example-F1, 25.40% Micro-F1 and 21.69% Macro-F1.
- CLEAR improves over the BCE baseline by on average 12.49% Example-F1, 11.89% Micro-F1 and 10.03% Macro-F1 across settings.
- Top-C anchor-point transition estimation (S-20) gives lower mean absolute error than the T-max and S-1 strategies in most dataset settings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CLEAR instead of the strongest baseline | CMLL · Image, Scene, Corel5K, Mirflickr, and NUS-WIDE (transition 0.2/0.2) | Example-F1: Image: 0.6433 -> 0.6730; Corel5K: 0.1240 -> 0.1237 | Mixed |
| Use CLEAR instead of the strongest baseline | CMLL · Image, Scene, Corel5K, Mirflickr, and NUS-WIDE (transition 0.2/0.5) | Example-F1: Mirflickr: 0.4403 -> 0.7267 | Helped |
| Use CLEAR instead of the strongest baseline | CMLL · Image, Scene, Corel5K, Mirflickr, and NUS-WIDE (transition 0.5/0.2) | Example-F1: Scene: 0.4484 -> 0.7043; NUS-WIDE: 0.0834 -> 0.0763 | Mixed |
| Replace the unbiased reconstruction loss with BCE loss (CLEAR-B) (ablation) | CLEAR · Image, Scene, and Mirflickr (transition 0.2/0.5) | F1 | Hurt |
| Remove label-correlation modeling by training independent class predictors (CLEAR-M) (ablation) | CLEAR · Image, Scene, and Mirflickr (transition 0.2/0.5) | F1 | Hurt |
| Increase the number of annotators | CLEAR · Image (transition 0.2/0.5) | Example-F1 | Helped |
| Estimate transition matrices by averaging the top-C confident crowdsourced labels (S-20) rather than T-max | CLEAR · Image, Scene, Corel5K, Mirflickr, and NUS-WIDE | Mean absolute error: Image 0.2/0.5: 0.31 -> 0.22 | Helped |
Surprising findings
- Crowdsourcing baselines such as MV and DoctorNet and the MLL baseline ML-KNN can perform worse than plain BCE when corruption is large, e.g., ML-KNN collapses on Image and Scene under (0.2, 0.5).
Limitations
- Assumes instance-independent, class-dependent transition matrices and that annotators make no profound mistakes.
- Assumes uniform annotator contribution when approximating the aggregated noisy posterior by averaging labels.
- Evaluated only on synthetically corrupted labels rather than real crowdsourced annotations.
Experimental setup
Datasets: Image Scene Corel5K Mirflickr NUS-WIDE
Benchmarks: Image Scene Corel5K Mirflickr NUS-WIDE
Metrics: example-F1 micro-F1 macro-F1 mean absolute error of transition matrix estimates
Baselines: BCE MV DoctorNet ML-KNN MPVAE PML-NI
Builds on: Multivariate Probit VAE (MPVAE) anchor point transition estimation DoctorNet PML-NI
Closest papers by meaning
- Multi-Instance Multi-Label Classification from Crowdsourced Labels (AAAI 2025 · similarity 0.90)
- Learning from Noisy Labels via Conditional Distributionally Robust Optimization (NeurIPS 2024 · similarity 0.88)
- Noisy Label Learning with Instance-Dependent Outliers: Identifiability via Crowd Wisdom (NeurIPS 2024 · similarity 0.88)
- Coupled Confusion Correction: Learning from Crowds with Sparse Annotations (AAAI 2024 · similarity 0.87)
- TLLC: Transfer Learning-based Label Completion for Crowdsourcing (ICML 2025 · similarity 0.87)
- Label Distribution Propagation-based Label Completion for Crowdsourcing (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.