Research map / Label-efficient and robust learning / Recognition and long-tail
Label Distribution Propagation-based Label Completion for Crowdsourcing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LDPLC, a label completion method for sparse crowdsourced label matrices that initializes label distributions via worker-similarity weighted voting and propagates them over instance neighbors using LLE-weighted neighbors.
Problem
Crowdsourced label matrices are very sparse because most workers annotate only a few instances, which degrades label integration algorithms; existing label completion (WSLC) ignores label correlations among similar instances.
Approach
Initialize label distributions for missing labels with worker-similarity weighted majority voting, then propagate them iteratively from neighboring instances (weights from locally linear embedding) so each missing label absorbs information from similar instances annotated by the same worker.
Key results
- On LabelMe, LDPLC raises MNLDP integration accuracy from 80.50% (WSLC) to 82.50%, and MV from 76.40% to 81.60%.
- Across 34 simulated datasets, LDPLC wins over WSLC significantly on 30 datasets for MV, DS, IWMV, GTIC and DEWSMV, and 27 for MNLDP, losing significantly on at most 1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace WSLC label completion with LDPLC | MV · LabelMe | accuracy: 76.40 -> 81.60 | Helped |
| Replace WSLC label completion with LDPLC | DS · LabelMe | accuracy: 76.70 -> 81.50 | Helped |
| Replace WSLC label completion with LDPLC | IWMV · LabelMe | accuracy: 76.30 -> 81.70 | Helped |
| Replace WSLC label completion with LDPLC | GTIC · LabelMe | accuracy: 76.40 -> 81.70 | Helped |
| Replace WSLC label completion with LDPLC | DEWSMV · LabelMe | accuracy: 76.50 -> 81.60 | Helped |
| Replace WSLC label completion with LDPLC | MNLDP · LabelMe | accuracy: 80.50 -> 82.50 | Helped |
| Replace WSLC label completion with LDPLC | Six label integration algorithms · 34 simulated datasets, uniform annotation quality | mean accuracy: 74.56 -> 83.51; 74.08 -> 83.57; 74.57 -> 83.54; 74.43 -> 83.45; 74.55 -> 83.54; 76.80 -> 83.99 | Helped |
| Replace WSLC label completion with LDPLC | Six label integration algorithms · 34 simulated datasets, Gaussian annotation quality | mean accuracy: 73.61 -> 82.72; 73.04 -> 82.63; 73.81 -> 82.73; 73.68 -> 82.66; 73.59 -> 82.70; 75.66 -> 83.13 | Helped |
Surprising findings
- On the simulated tic-tac-toe dataset, LDPLC is slightly worse than WSLC for MNLDP (76.75 vs 78.66 for WSLC).
Limitations
- Performance degrades in high-noise crowdsourcing scenarios because inaccurate initial label distributions propagate errors.
- Evaluated mainly on a simulated setting with ~95% missing labels; real-world evaluation limited to a few datasets.
Experimental setup
Datasets: LabelMe Ruters Leaves 34 simulated datasets from CEKA
Benchmarks: LabelMe Ruters Leaves 34 simulated CEKA datasets
Metrics: integration accuracy win/tie/lose with corrected paired t-test
Baselines: WSLC PMF PMF-TLC
Builds on: WSLC MNLDP PMF label completion
Closest papers by meaning
- TLLC: Transfer Learning-based Label Completion for Crowdsourcing (ICML 2025 · similarity 0.92)
- Unbiased Multi-Label Learning from Crowdsourced Annotations (ICML 2024 · similarity 0.86)
- Efficient Online Crowdsourcing with Complex Annotations (AAAI 2024 · similarity 0.86)
- Label Aggregation for Composite Crowd Tasks by Worker Ability Constraint Satisfaction (AAAI 2025 · similarity 0.85)
- KFNN: K-Free Nearest Neighbor For Crowdsourcing (NeurIPS 2024 · similarity 0.85)
- IWBVT: Instance Weighting-based Bias-Variance Trade-off for Crowdsourcing (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.