Research map / Label-efficient and robust learning / Recognition and long-tail
Mixture of Experts Based Multi-Task Supervise Learning from Crowds
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MMLC, a mixture-of-experts multi-task supervised learning-from-crowds model that models worker behavior at the item-feature level, with two aggregation strategies (oracle worker finding and data filling) for inferring ground truth.
Problem
Learning from crowdsourced noisy labels: existing worker behavior models (confusion matrices, ground-truth latent variables) fail to capture how worker behavior varies across item features, hurting truth inference.
Approach
Replace ground-truth latent variables with a multi-task model in which each worker is a task; a mixture-of-experts over item features yields a worker spectral space, where worker projections are clustered (KDE/mean/median) to find an oracle worker whose labels serve as ground truth (MMLC-owf), or the model fills missing crowd labels for use with existing aggregators (MMLC-df).
Key results
- MMLC-owf reaches 74.31% on Text and 79.14% on Music, the best among compared learning-from-crowds methods; on LabelMe it is 81.74% vs 82.14% for CrowdAR.
- MMLC-df improves accuracy over the original data in all 24 scenarios (8 aggregators x 3 datasets), with gains up to +7.58 on Music with MV.
- With ground-truth-optimized projections, the 'Truth' upper bound reaches 96.32% (LabelMe), 91.25% (Text), and 92.97% (Music).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MMLC-owf instead of CrowdAR | Learning-from-crowds truth inference · LableMe | accuracy | Hurt |
| Use MMLC-owf instead of the strongest baseline | Learning-from-crowds truth inference · Text | accuracy | Helped |
| Use MMLC-owf instead of the strongest baseline | Learning-from-crowds truth inference · Music | accuracy | Helped |
| Fill sparse crowdsourced data using MMLC-df | Learning-from-crowds methods on LableMe · LableMe | accuracy: +3.89$\pm$0.17 (MV); +1.72$\pm$0.20 (DS); +3.38$\pm$0.21 (FDS); +2.48$\pm$0.36 (HDS); +4.81$\pm$0.21 (Max-MIG); +1.73$\p | Helped |
| Fill sparse crowdsourced data using MMLC-df | Learning-from-crowds methods on Text · Text | accuracy: +3.35$\pm$0.31 (MV); +4.02$\pm$0.26 (DS); +3.26$\pm$0.35 (FDS); +2.18$\pm$0.19 (HDS); +3.88$\pm$0.51 (Max-MIG); +2.16$\p | Helped |
| Fill sparse crowdsourced data using MMLC-df | Learning-from-crowds methods on Music · Music | accuracy: +7.58$\pm$0.45 (MV); +2.41$\pm$0.41 (DS); +1.57$\pm$0.22 (FDS); +0.12$\pm$0.54 (HDS); +4.77$\pm$0.41 (Max-MIG); +6.37$\p | Helped |
| Use KDE to estimate the oracle worker projection rather than the other tested clustering methods | MMLC-owf · LableMe, Text, and Music | accuracy | Helped |
| Increase the maximum annotation redundancy | MMLC-owf · LableMe, Text, and Music | accuracy | Helped |
| Increase data-filling density | MMLC-df with statistical learning methods · LableMe | accuracy | Mixed |
| Increase data-filling density | MMLC-df with statistical learning methods · Text | accuracy | No clear effect |
| Increase data-filling density | MMLC-df with statistical learning methods · Music | accuracy | Mixed |
Surprising findings
- MMLC-df is reported to improve every scenario, while G_MV, G_IRT, and TDG4Crowd improve only in 50%, 62.5%, and 75% of them.
Limitations
- MMLC-owf may underfit on datasets with low redundancy.
- Oracle worker projection still deviates from the theoretical upper bound, and clustering choice matters.
- Deep-learning variants are slow for large filled datasets, so density analysis used only statistical aggregators.
Experimental setup
Datasets: LabelMe Text Music
Benchmarks: LabelMe Text Music
Metrics: accuracy
Baselines: MV DS HDS FDS Max-MIG CoNAL CrowdAR G_MV G_IRT TDG4Crowd
Builds on: Max-MIG CoNAL CrowdAR MMoE TDG4Crowd
Closest papers by meaning
- Label Aggregation for Composite Crowd Tasks by Worker Ability Constraint Satisfaction (AAAI 2025 · similarity 0.84)
- TLLC: Transfer Learning-based Label Completion for Crowdsourcing (ICML 2025 · similarity 0.84)
- Label Distribution Propagation-based Label Completion for Crowdsourcing (ICML 2025 · similarity 0.84)
- KFNN: K-Free Nearest Neighbor For Crowdsourcing (NeurIPS 2024 · similarity 0.82)
- Unbiased Multi-Label Learning from Crowdsourced Annotations (ICML 2024 · similarity 0.82)
- IWBVT: Instance Weighting-based Bias-Variance Trade-off for Crowdsourcing (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.