Research map / Label-efficient and robust learning / Noisy labels
Deep Learning from Crowds
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with an annotator-specific matrix crowd layer rather than the EM approach | CNN · Dogs vs. Cats | accuracy: 80.184 ($\pm$ 1.454) -> 80.265 ($\pm$ 1.230) | No clear effect |
| Train with an annotator-specific matrix crowd layer rather than the EM approach | CNN · LabelMe (MTurk) | accuracy: 82.677 ($\pm$ 0.981) -> 83.151 ($\pm$ 0.877) | Helped |
| Use per-class weights instead of a matrix crowd mapping (ablation) | DL-CL · Dogs vs. Cats | accuracy: 79.534 ($\pm$ 1.064) vs. 80.265 ($\pm$ 1.230) | Hurt |
| Use per-class weights instead of a matrix crowd mapping (ablation) | DL-CL · LabelMe (MTurk) | accuracy: 81.051 ($\pm$ 0.899) vs. 83.151 ($\pm$ 0.877) | Hurt |
| Use per-class weights and biases instead of a matrix crowd mapping (ablation) | DL-CL · Dogs vs. Cats | accuracy: 79.688 ($\pm$ 1.406) vs. 80.265 ($\pm$ 1.230) | Hurt |
| Use per-class weights and biases instead of a matrix crowd mapping (ablation) | DL-CL · LabelMe (MTurk) | accuracy: 81.886 ($\pm$ 0.893) vs. 83.151 ($\pm$ 0.877) | Hurt |
| Add a per-annotator bias instead of using scale only (ablation) | DL-CL · MovieReviews (MTurk) | MAE: 1.228 ($\pm$ 0.041) -> 1.130 ($\pm$ 0.025) | Helped |
| Add a per-annotator bias to the scale-and-bias crowd layer (ablation) | DL-CL · MovieReviews (MTurk) | R^2: 30.560 ($\pm$ 4.101) -> 37.086 ($\pm$ 2.407) | Helped |
| Use per-annotator bias only rather than scale only (ablation) | DL-CL · MovieReviews (MTurk) | RMSE: 1.508 ($\pm$ 0.044) -> 1.411 ($\pm$ 0.028) | Helped |
| Use the matrix crowd layer rather than the best reported non-CL neural baseline | CNN-GRU · CoNLL-2003 NER (MTurk) | F1: 0.575 ($\pm$ 0.008) -> 0.624 ($\pm$ 0.010) | Helped |
| Use per-class weights and biases instead of a matrix crowd mapping (ablation) | DL-CL · CoNLL-2003 NER (MTurk) | F1: 0.606 ($\pm$ 0.007) vs. 0.624 ($\pm$ 0.010) | Hurt |
| Use per-class weights instead of a matrix crowd mapping (ablation) | DL-CL · CoNLL-2003 NER (MTurk) | F1: 0.566 ($\pm$ 0.016) vs. 0.624 ($\pm$ 0.010) | Hurt |
Closest papers by meaning
- Semi-Supervised Learning From Crowds Using Deep Generative Models (AAAI 2018 · similarity 0.90)
- Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts (NeurIPS 2020 · similarity 0.87)
- Learning from Crowds by Modeling Common Confusions (AAAI 2021 · similarity 0.87)
- Coupled Confusion Correction: Learning from Crowds with Sparse Annotations (AAAI 2024 · similarity 0.86)
- KFNN: K-Free Nearest Neighbor For Crowdsourcing (NeurIPS 2024 · similarity 0.85)
- Unbiased Multi-Label Learning from Crowdsourced Annotations (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.