Research map / Label-efficient and robust learning / Noisy labels
Collaborative Refining for Learning from Inaccurate Labels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CRL, a collaborative refining framework that uses annotator agreement to refine multiple sets of inaccurate labels (LRD for disagreeing samples, RUS for agreeing samples) and trains a lightweight MLP on the refined data.
Problem
Learning binary classifiers from multiple sets of inaccurate labels produced by low-cost annotators (e.g., rule-based annotators), where prior work focuses on aggregating labels rather than refining the data.
Approach
Partition samples by annotator agreement. For disagreeing samples, pick the label whose submodel has the smallest loss (theoretically justified under class-conditional noise with diagonal transition entries above 0.5). For agreeing samples, select trustworthy ones with a robust loss-based criterion built from a smooth log-Taylor function over historical epochs and a confidence lower bound. A final
Key results
- On class-dependent noise benchmarks, CRL reaches 0.855 AUC on AgNews vs 0.833 for the best baseline (WeaSEL) and achieves the best average across the 13 benchmarks.
- On real-world data, CRL scores 0.866 AUC on CIFAR-10N (vs 0.851 for WeaSEL) and 0.753 on Sentiment (vs 0.744 for HE_A).
- Ablation on class-dependent noise: baseline 0.634 average AUC rises to 0.855 with LRD and RUS combined; LRD alone gives 0.832 average.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply CRL with collaborative refining (LRD and RUS) instead of majority-vote label aggregation (ablation) | MLP trained on inaccurate labels · Ablation datasets (Avg.) | AUC: 0.643 -> 0.814 | Helped |
| Add RUS while using majority voting instead of LRD (ablation) | CRL ablation · Ablation datasets (Avg.) | AUC: 0.643 -> 0.749 | Helped |
| Add LRD while using majority voting instead of RUS (ablation) | CRL ablation · Ablation datasets (Avg.) | AUC: 0.643 -> 0.778 | Helped |
| Replace RUS criterion with naive mean-loss selection (ablation) | CRL with LRD · Ablation datasets (Avg.) | AUC: 0.814 -> 0.796 | Hurt |
| Use LRD and RUS together rather than LRD with naive RUS selection (ablation) | CRL · Ablation datasets (Avg.) | AUC: 0.796 -> 0.814 | Helped |
| Apply CRL rather than the strongest listed baseline | MLP trained on class-dependent noisy labels · Benchmark datasets (class-dependent noise) | AUC | Helped |
| Apply CRL rather than the strongest listed baseline | MLP trained on real-world noisy labels · Sentiment and CIFAR-10N | AUC | Helped |
Surprising findings
- Naive loss-mean selection (N-RUS) gives smaller gains than the robust RUS criterion, and LRD gives larger gains than RUS alone.
- Labels from LRD frozen at step 100 still reach 0.835 average AUC, close to 0.845 when frozen at step 500, suggesting flexibility in when to freeze refined labels.
Limitations
- Theory and method restricted to binary classification; multi-class extension needs adaptation (e.g., when all labels are wrong).
- Results are averaged without error bars or statistical significance tests.
- Code and data not released at submission; compute resources not reported.
Experimental setup
Datasets: AgNews 20News IMDb Yelp Amazon Diabetes Backdoor Campaign Waveform Celeba SVHN F-MNIST
Benchmarks: AgNews 20News IMDb Yelp Amazon Diabetes Backdoor Campaign Waveform Celeba SVHN F-MNIST
Metrics: AUC
Baselines: Single NN-Mjv HE_A HE_M CL DN NN-EBCC NN-IBCC WeaSEL SLF CoNAL ADMoE
Builds on: small-loss criterion Crowd Layer DoctorNet CoNAL SLF ADMoE
Closest papers by meaning
- ULAREF: A Unified Label Refinement Framework for Learning with Inaccurate Supervision (ICML 2024 · similarity 0.87)
- Coupled Confusion Correction: Learning from Crowds with Sparse Annotations (AAAI 2024 · similarity 0.86)
- Generative Calibration of Inaccurate Annotation for Label Distribution Learning (AAAI 2024 · similarity 0.85)
- Noisy Label Learning with Instance-Dependent Outliers: Identifiability via Crowd Wisdom (NeurIPS 2024 · similarity 0.85)
- Predicting Label Distribution from Ternary Labels (NeurIPS 2024 · similarity 0.85)
- Addressing Multi-Label Learning with Partial Labels: From Sample Selection to Label Selection (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.