Research map / Label-efficient and robust learning / Recognition and long-tail
Multi-Instance Multi-Label Classification from Crowdsourced Labels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes cross-label transition matrices and unbiased risk estimators for crowdsourced multi-instance multi-label classification (CMIML), with aggregated bag-level and instance-level variants and a generalization bound.
Problem
Learning multi-instance multi-label classifiers from noisy labels collected from multiple crowdsourced annotators, where annotation errors are correlated across labels rather than independent per label.
Approach
Models annotation noise as cross-label transitions (class-dependent transition matrices over label pairs across classes) and derives the first unbiased risk estimator for CMIML, simplified via aggregating crowd labels and transition matrices into a bag-level loss with a label-confidence vector estimated from anchor points.
Key results
- Under transition setting (0.5,0.5), Ours reaches Macro-F1 0.6204 on Scene and 0.7823 on Reuters, outperforming baselines such as MIML-LLMC (0.3952 and 0.8742 respectively are reported for MIML-LLMC, so Ours is not best on Reuters Macro-F1 in that table).
- On PyrFur under transition setting (0.3,0.5), Ours achieves AvgRec 0.5048 versus MIMLSVM's 0.1119, which the authors describe as about 4.5 times the second-best model.
- The paper reports that Ours achieves the lowest label-confidence error as annotators grow; with 50 annotators error drops to 0.11 on Scene and below 0.04 on Reuters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed CMIML unbiased loss instead of BCE loss (ablation) | Ours · Scene, Reuters, AzoVin, GeoSul, HalMar, and PyrFur | Macro-F1: Scene: 0.3967 -> 0.6204; Reuters: 0.7481 -> 0.7823; AzoVin: 0.0542 -> 0.0731; GeoSul: 0.0305 -> 0.0746; HalMar: 0.1125 - | Mixed |
| Use the proposed CMIML unbiased loss instead of BCE loss (ablation) | Ours · Scene, Reuters, AzoVin, GeoSul, HalMar, and PyrFur | AvgRec: Scene: 0.5477 -> 0.6492; Reuters: 0.9105 -> 0.9723; AzoVin: 0.0815 -> 0.4759; GeoSul: 0.1108 -> 0.4959; HalMar: 0.1910 - | Helped |
| Use the proposed CMIML unbiased loss instead of BCE loss (ablation) | Ours · Scene, Reuters, AzoVin, GeoSul, HalMar, and PyrFur | Coverage: Scene: 2.48 -> 1.47; Reuters: 1.34 -> 0.31; AzoVin: 59.18 -> 44.56; GeoSul: 46.78 -> 37.01; HalMar: 40.01 -> 27.03; PyrF | Mixed |
| Apply the proposed CMIML method rather than the strongest baseline | Ours · Scene, Reuters, AzoVin, GeoSul, HalMar, and PyrFur | Macro-F1 | Mixed |
| Apply the proposed CMIML method rather than the strongest baseline | Ours · Scene, Reuters, AzoVin, GeoSul, HalMar, and PyrFur | Coverage | Mixed |
| Apply the proposed CMIML method rather than the strongest baseline | Ours · Scene, Reuters, AzoVin, GeoSul, HalMar, and PyrFur | AvgRec | Helped |
Surprising findings
- On several datasets (e.g., Reuters Macro-F1 and AzoVin/GeoSul Macro-F1 in some settings), MIML-LLMC outperforms the proposed method on Macro-F1.
Limitations
- Experiments use simulated crowdsourced labels generated from assumed transition matrices rather than real crowdsourced annotations.
- Instance-level URE is only derived theoretically and not implemented in experiments.
- Transition matrix estimation assumes anchor points and uniform annotator contributions.
Experimental setup
Datasets: Scene Reuters AzoVin GeoSul HalMar PyrFur
Benchmarks: Scene Reuters AzoVin GeoSul HalMar PyrFur
Metrics: Macro-F1 Coverage AvgRec
Baselines: MIMLSVM KiSar DeepMIML MIML-LLMC PLAIN Ours-BCE
Builds on: unbiased risk estimator single-label transition matrix methods (Patrini et al. 2017) anchor point estimation (Liu and Tao 2015) Unbiased Multi-Label Learning from Crowdsourced Annotations (Xia et al. 2024)
Closest papers by meaning
- Unbiased Multi-Label Learning from Crowdsourced Annotations (ICML 2024 · similarity 0.90)
- T-MDML: Triplet-based Multiple Distance Metric Learning for Multi-Instance Multi-Label Classification with Label Correlation (AAAI 2025 · similarity 0.87)
- Noisy Label Learning with Instance-Dependent Outliers: Identifiability via Crowd Wisdom (NeurIPS 2024 · similarity 0.85)
- Distilling Reliable Knowledge for Instance-Dependent Partial Label Learning (AAAI 2024 · similarity 0.85)
- TLLC: Transfer Learning-based Label Completion for Crowdsourcing (ICML 2025 · similarity 0.85)
- Multi-View Multi-Label Classification via View-Label Matching Selection (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.