Research map / Label-efficient and robust learning / Recognition and long-tail
Lean Multiclass Crowdsourcing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace majority voting with Flat Multinomial worker-skill aggregation | Bluebird crowdsourcing · Bluebird dataset | label error rate: 24.07 -> 11.11 | Helped |
| Use computer vision probabilities in Flat Multinomial aggregation (ablation) | Flat Multinomial · Bluebird dataset | label error rate: 11.11 -> 10.19 | Helped |
| Use worker-skill aggregation instead of majority voting | NABirds crowdsourcing · NABirds, 69 species | annotation count to reach the same error: 5.4× decrease | Helped |
| Add computer vision probabilities to the single binomial model (ablation) | NABirds crowdsourcing · NABirds, 69 species, Combined | error: 25% decrease | Helped |
| Initialize global priors from a small held-out set with ground-truth labels (ablation) | NABirds worker-skill models · NABirds, 69 species, Combined-Prior | error: 0.03; 79% decrease from majority vote | Helped |
| Model dependence between labels (ablation) | Flat single binomial worker model · iNaturalist birds, 30 species | error: 59% decrease | Helped |
| Model dependence between labels (ablation) | Flat multinomial worker model · iNaturalist birds, 30 species | error: 85% decrease | Helped |
| Use a taxonomic rather than flat worker-skill model (ablation) | Single binomial worker model · iNaturalist birds, 30 species | error | Hurt |
| Use a taxonomic rather than flat worker-skill model (ablation) | Multinomial worker model · iNaturalist birds, 30 species | error: 36% decrease | Helped |
| Add dependence modeling to taxonomic worker-skill models (ablation) | Taxonomic multinomial worker model · iNaturalist birds, 30 species | error: 90% decrease over majority vote | Helped |
| Use the taxonomic multinomial model with dependence modeling instead of majority voting | iNaturalist crowdsourcing · iNaturalist birds, 30 species | error: 90% decrease; 28 total errors | Helped |
Closest papers by meaning
- Lean Crowdsourcing: Combining Humans and Machines in an Online System (CVPR 2017 · similarity 0.93)
- Learning From Noisy Singly-labeled Data (ICLR 2018 · similarity 0.89)
- Unbiased Multi-Label Learning from Crowdsourced Annotations (ICML 2024 · similarity 0.86)
- Coupled Confusion Correction: Learning from Crowds with Sparse Annotations (AAAI 2024 · similarity 0.86)
- Efficient Online Crowdsourcing with Complex Annotations (AAAI 2024 · similarity 0.86)
- Doomed to Re-Annotate, Forever: The ImageNet Story (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.