Research map / Label-efficient and robust learning / Noisy labels
Label Aggregation for Composite Crowd Tasks by Worker Ability Constraint Satisfaction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes label aggregation for Composite Crowd Tasks (CCTs) that link multiple heterogeneous or multi-object-set crowd tasks through shared worker-ability constraints, relaxing the constraint into a regularized multi-task objective over GLAD, CrowdBT, HBTL, and kernel-based similarity models.
Problem
Crowd label aggregation has focused on single crowd tasks with one object set and one question type, leaving composite tasks mixing heterogeneous question types or multiple object sets without methods that share worker-ability information across tasks.
Approach
Treat worker abilities shared across tasks in a composite crowd task as a soft equality (or rank-order) constraint, relaxed into a regularizer added to the sum of per-task backbone aggregation losses, so that each task's aggregation benefits from the other tasks' labels.
Key results
- On the Case 2 pairwise similarity datasets, the CCT models with the relaxed constraints improve over Student-t and Gaussian kernel baselines (e.g., about 0.5459 vs 0.5458 type gains; reported 0.5507 vs 0.5458 for cheat with Euclidean constraint).
- On the categorization and preference group, Task 2 preference accuracy rises from about 0.435 to 0.644 for MO and from 0.364 to 0.574 for CI with the Euclidean constraint.
- Improvements are statistically significant (Wilcoxon, p<0.05) in most settings, and the four constraint variants perform comparably.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Bridge worker abilities across composite crowd tasks | CCT label aggregation · Case 1 and Case 2 pairwise preference and similarity tasks | pairwise comparison accuracy: 0.6906 ± 0.0180 -> 0.6953 ± 0.0178*; 0.5259 ± 0.0292 -> 0.5336 ± 0.0281* | Helped |
| Bridge worker abilities between categorization and preference tasks | CCT label aggregation · MOON, CIRCLE, CLASS2, CLASS3, BLOB3 | pairwise preference accuracy: 0.435 -> 0.644 ± 0.127*; 0.364 -> 0.586 ± 0.142*; 0.261 -> 0.536 ± 0.131*; 0.462 -> 0.537 ± 0.131*; 0.455 -> 0.502 ± 0.1 | Helped |
| Use Euclidean, KL, L1, or rank worker-ability constraints instead of another constraint variant (ablation) | CCT label aggregation · Pairwise comparison datasets | pairwise comparison accuracy | No clear effect |
Surprising findings
- The four constraint variants (Euclidean, KL, L1, rank-based) yield comparable results, with no clear winner.
Limitations
- Benefit depends on the composite tasks having comparable difficulty or on one task's worker abilities being easy to estimate.
- The rank-based constraint can be harmful if worker abilities are inconsistent across tasks.
- Only two-task settings are evaluated in experiments; more tasks are only formulated.
Experimental setup
Datasets: cheat and meeting opinion datasets from Baba, Li, and Kashima 2020 (opinions only; labels recollected) Composite crowd task datasets collected by the authors (categorization and preference)
Benchmarks: cheat/meeting pairwise preference and similarity crowd datasets categorization and preference dataset group (MO, CI, CL2, CL3, BL3)
Metrics: category accuracy pairwise preference accuracy pairwise similarity accuracy
Baselines: GLAD CROWDBT HBTL Gaussian kernel similarity model Student-t kernel similarity model
Builds on: GLAD CrowdBT HBTL Dawid-Skene
Closest papers by meaning
- Efficient Online Crowdsourcing with Complex Annotations (AAAI 2024 · similarity 0.85)
- Label Distribution Propagation-based Label Completion for Crowdsourcing (ICML 2025 · similarity 0.85)
- TLLC: Transfer Learning-based Label Completion for Crowdsourcing (ICML 2025 · similarity 0.85)
- Mixture of Experts Based Multi-Task Supervise Learning from Crowds (AAAI 2025 · similarity 0.84)
- IWBVT: Instance Weighting-based Bias-Variance Trade-off for Crowdsourcing (NeurIPS 2024 · similarity 0.84)
- Unbiased Multi-Label Learning from Crowdsourced Annotations (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.