Research map / Label-efficient and robust learning / Noisy labels
IWBVT: Instance Weighting-based Bias-Variance Trade-off for Crowdsourcing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IWBVT, a universal post-processing approach for crowdsourcing that combines a complementary-set-and-entropy instance weighting with bias-variance-trade-off probabilistic loss regressions to improve model quality on top of label integration and noise correction algorithms.
Problem
Label integration and noise correction algorithms improve label quality in crowdsourcing but yield only modest gains in the quality of models trained on the integrated labels, because intractable instances with ambiguous attributes hurt model learning.
Approach
A new instance weighting method based on P(ŷ|L) and the entropy of the complementary distribution (handles complex multi-class label distributions that prior weighting methods cannot distinguish), followed by bias-variance trade-off via per-class probabilistic loss regressions added to the classifier's scores.
Key results
- Averaged over 34 simulated datasets, MV with IWBVT reaches 79.03% model quality vs 75.45% for plain MV; AVNC improves from 78.57% to 80.77%.
- IWBVT yields significant wins over losses for all seven base algorithms (e.g., 17 wins/12 ties/5 losses for MV).
- On real-world Leaves and Income, IWBVT improves model quality of label integration and noise correction methods, with instance weighting more effective on Leaves and bias-variance trade-off more effective on Income.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply IWBVT post-processing | MV · 34 simulated datasets (average) | model quality (%): 75.45 -> 79.03 | Helped |
| Apply IWBVT post-processing | IWMV · 34 simulated datasets (average) | model quality (%): 75.31 -> 78.94 | Helped |
| Apply IWBVT post-processing | LAWMV · 34 simulated datasets (average) | model quality (%): 78.00 -> 80.22 | Helped |
| Apply IWBVT post-processing | MNLDP · 34 simulated datasets (average) | model quality (%): 76.96 -> 80.24 | Helped |
| Apply IWBVT post-processing | AVNC · 34 simulated datasets (average) | model quality (%): 78.57 -> 80.77 | Helped |
| Apply IWBVT post-processing | MVNC · 34 simulated datasets (average) | model quality (%): 77.16 -> 80.44 | Helped |
| Apply IWBVT post-processing | NWVNC · 34 simulated datasets (average) | model quality (%): 76.42 -> 80.54 | Helped |
| Apply IWBVT post-processing | MV · Leaves | model quality (%): 59.75 -> 62.28 | Helped |
| Apply IWBVT post-processing | MV · Income | model quality (%): 69.22 -> 71.35 | Helped |
| Add instance weighting only (ablation) | MV · Leaves | model quality (%): 59.91 -> 62.23 | Helped |
| Add bias-variance trade-off only (ablation) | MV · Leaves | model quality (%): 59.91 -> 60.18 | Helped |
| Add instance weighting only (ablation) | MV · Income | model quality (%): 69.05 -> 70.03 | Helped |
Surprising findings
- Several state-of-the-art label integration and noise correction algorithms yield lower model quality than plain MV on real-world datasets.
- IWBVT degrades model quality on a few datasets (e.g., labor, lymph) with attributes of very different magnitude, attributed to linear regression's lack of robustness.
Limitations
- Not robust on datasets with anomalous, high-magnitude numerical attributes (e.g., labor, lymph), where linear regression degrades performance.
- Relies on linear regression and Naive Bayes as the regression and classification models; results depend on these choices.
- Evaluation relies on simulated workers with a fixed normal label-quality distribution and only two real-world datasets.
Experimental setup
Datasets: 34 simulated datasets from CEKA Leaves (CEKA) Income (CEKA)
Benchmarks: 34 simulated CEKA datasets Leaves Income
Metrics: model quality (accuracy %) win/tie/loss with corrected paired t-test bias and variance
Baselines: MV IWMV LAWMV MNLDP AVNC MVNC NWVNC
Builds on: bias-variance decomposition label augmented and weighted majority voting multiple noisy label distribution propagation adaptive voting noise correction
Closest papers by meaning
- Unbiased Multi-Label Learning from Crowdsourced Annotations (ICML 2024 · similarity 0.85)
- Efficient Online Crowdsourcing with Complex Annotations (AAAI 2024 · similarity 0.85)
- Label Distribution Propagation-based Label Completion for Crowdsourcing (ICML 2025 · similarity 0.84)
- Label Aggregation for Composite Crowd Tasks by Worker Ability Constraint Satisfaction (AAAI 2025 · similarity 0.84)
- KFNN: K-Free Nearest Neighbor For Crowdsourcing (NeurIPS 2024 · similarity 0.84)
- Multi-Instance Multi-Label Classification from Crowdsourced Labels (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.