Research map / Label-efficient and robust learning / Noisy labels
Coupled Confusion Correction: Learning from Crowds with Sparse Annotations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Coupled Confusion Correction (CCC), where two DNNs distill meta sets for each other and use meta-learning with K-Means-clustered annotator groups to correct sparse annotators' confusion matrices, plus a Beta-distribution scheme for more realistic synthetic crowd annotations.
Problem
Learning from crowds with sparse annotations: annotators who label few instances have poorly learned confusion matrices (ASCPs), degrading the classifier; existing synthetic crowd-sourcing benchmarks also ignore few-label annotators.
Approach
Two simultaneously trained models distill clean meta sets for each other; confusion matrices are corrected via bi-level meta-learning using these meta sets, with annotators clustered by K-Means into groups that share a correction matrix, so sparse annotators borrow supervision from their group and meta set. Synthetic annotations are generated with Beta-distributed annotator selection to mimic few-
Key results
- On synthetic CIFAR-10 IND-I, CCC reaches 80.16% best test accuracy vs 74.67% for UnionNet (best baseline).
- On real-world CIFAR-10N, CCC reaches 86.36% best accuracy vs 83.68% for DoctorNet; on MUSIC 76.22% vs 75.33% for Max-MIG.
- In the sparsest setting (average 1 label per instance), CCC improves best accuracy by 17.15% over CrowdLayer.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CCC's coupled meta-set correction and annotator-grouping to CrowdLayer | CIFAR-10 and Fashion-MNIST synthetic crowd-label training · Synthetic independent and correlated confusion cases | test accuracy | Helped |
| Use CCC instead of the strongest baseline | CIFAR-10N · CIFAR-10N | test accuracy (best and last): ↑ 2.68; ↑ 2.80 | Helped |
| Use CCC instead of the strongest baseline | MUSIC · MUSIC | test accuracy (best and last): ↑ 0.89; ↑ 1.22 | Helped |
| Use CCC instead of the strongest baseline | LabelMe · LabelMe | test accuracy (best and last): ↑ 0.19; ↓ 0.06 | Mixed |
| Increase the average number of annotations per instance (ablation) | CCC on CIFAR-10 synthetic crowd labels · CIFAR-10 with varying annotation sparsity | test accuracy: 17.15% on best and 32.95% on last | Helped |
| Vary the number of annotator groups G (ablation) | CCC · CIFAR-10-IND-I | test accuracy | Mixed |
| Train with crowd labels rather than clean ground-truth labels | CCC · CIFAR-10N | test accuracy | Hurt |
Surprising findings
- CCC results on CIFAR-10N are comparable to training with clean labels, per the authors.
Limitations
- Bi-level optimization is time-consuming because of Hessian computation.
- Number of annotator groups G is a hyperparameter that is hard to identify without true labels.
- Relies on the assumption that many annotators share similar expertise.
Experimental setup
Datasets: CIFAR-10 Fashion-MNIST LabelMe CIFAR-10N MUSIC
Benchmarks: CIFAR-10 synthetic (IND/COR) Fashion-MNIST synthetic (IND/COR) LabelMe CIFAR-10N MUSIC
Metrics: test accuracy (best and last epoch)
Baselines: MajorVote CrowdLayer DoctorNet Max-MIG CoNAL UnionNet
Builds on: CrowdLayer Max-MIG UnionNet FaMUS MLC
Closest papers by meaning
- Noisy Label Learning with Instance-Dependent Outliers: Identifiability via Crowd Wisdom (NeurIPS 2024 · similarity 0.88)
- Unbiased Multi-Label Learning from Crowdsourced Annotations (ICML 2024 · similarity 0.87)
- Efficient Online Crowdsourcing with Complex Annotations (AAAI 2024 · similarity 0.86)
- Collaborative Refining for Learning from Inaccurate Labels (NeurIPS 2024 · similarity 0.86)
- TLLC: Transfer Learning-based Label Completion for Crowdsourcing (ICML 2025 · similarity 0.86)
- Learning from Noisy Labels via Conditional Distributionally Robust Optimization (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.