Research map / Label-efficient and robust learning / Recognition and long-tail
Efficient Online Crowdsourcing with Complex Annotations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes online accuracy estimation for complex crowdsourcing annotations using average pairwise similarity partitioned by reported label type (POAK), with a conditional Anna Karenina theorem and an IRT-based variant, to decide when to stop collecting labels.
Problem
Online crowdsourcing with complex annotations (bounding boxes, taxonomy paths, keypoints) requires deciding on-the-fly whether to collect another annotation per item, but existing truth discovery methods assume offline data and categorical labels.
Approach
Estimate each labeler's accuracy from average similarity to other labelers, conditioned on the reported label type (partition of label space), supported by a conditional Anna Karenina theorem; an IRT parameterization reduces latent variables. The estimate drives a stop-or-ask decision per item.
Key results
- POAK-Weight dominates other methods in RAUC across the four Meta datasets, outperforming OAK-Weight, SAD, and Uniform.
- On the Keypoint dataset, POAK matches the baseline accuracy using only 81% of labels.
- POAKi and POAK+IRT outperform POAK, especially when training samples are small.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use confidence-based POAK-Weight or OAK-Weight instead of SAD or Uniform | Online crowdsourcing methods · Keypoint, Taxonomy, TreePath, BoundingBox | RAUC | Helped |
| Use partition-based POAK instead of OAK (ablation) | Online crowdsourcing methods · Keypoint, Taxonomy, TreePath, BoundingBox | RAUC | Helped |
| Use averaging or majority voting instead of selection | Crowdsourcing aggregation · Most experimental settings | RAUC | Helped |
| Add POAKi calibration to POAK (ablation) | POAK · Cost-quality evaluation | RAUC | Helped |
| Add IRT-based regularization to POAKi (ablation) | POAKi · Cost-quality evaluation | RAUC | Helped |
| Use auditor labels for calibration (ablation) | POAK · Datasets with auditor labels | accuracy | Helped |
| Add a second decision point using the same threshold (ablation) | Online crowdsourcing methods · Taxonomy and BoundingBox | RAUC | Mixed |
| Use POAK instead of a random label-count decision | Keypoint · Keypoint | labels used at on-par accuracy: 81% | Helped |
Surprising findings
- Averaging or majority voting outperforms selection-based aggregation in most settings.
- Using two decision points with the same threshold can hurt performance on the BoundingBox dataset.
Limitations
- Datasets are proprietary and cannot be shared, limiting reproducibility.
- Partitioning of label types is determined externally with domain-specific heuristics.
- Theorem proven only for categorical labels; extension to complex labels is empirical.
Experimental setup
Datasets: Keypoint Taxonomy TreePath BoundingBox Age
Benchmarks: Keypoint Taxonomy TreePath BoundingBox
Metrics: RAUC (relative area under cost-accuracy curve)
Baselines: SAD Uniform BAU OAK-Weight
Builds on: Proximity-based Truth Discovery (PTD) Dawid-Skene model Item Response Theory CLARA
Closest papers by meaning
- Coupled Confusion Correction: Learning from Crowds with Sparse Annotations (AAAI 2024 · similarity 0.86)
- Label Distribution Propagation-based Label Completion for Crowdsourcing (ICML 2025 · similarity 0.86)
- Unbiased Multi-Label Learning from Crowdsourced Annotations (ICML 2024 · similarity 0.85)
- Label Aggregation for Composite Crowd Tasks by Worker Ability Constraint Satisfaction (AAAI 2025 · similarity 0.85)
- IWBVT: Instance Weighting-based Bias-Variance Trade-off for Crowdsourcing (NeurIPS 2024 · similarity 0.85)
- Evaluating LLM-contaminated Crowdsourcing Data Without Ground Truth (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.