Research map / Label-efficient and robust learning / Recognition and long-tail
Soft-Label Integration for Robust Toxicity Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a bi-level optimization framework that learns soft-label weights over crowdsourced toxicity annotations and optimizes them with GroupDRO, training a RoBERTa-large classifier via ERM in the inner loop.
Problem
Toxicity classification from multiple noisy crowdsourced/LLM annotations is hurt by single-annotator labels and by ERM exploiting spurious correlations, which degrades worst-group and out-of-distribution accuracy.
Approach
Learn instance-dependent soft-label weights over candidate annotator labels with a neural weight estimator, optimizing them in the outer loop by a GroupDRO risk on a small clean validation set, while the classifier is trained by ERM with the soft labels in the inner loop.
Key results
- On Q-A, the method reaches 78.41% average and 69.44% worst-group accuracy, above the best baseline Vanilla Soft Label (74.81%, 67.68%).
- On R-A, the method reaches 89.80% average and 77.82% worst-group accuracy, versus 85.52% and 62.57% for Vanilla Soft Label.
- Pre-training and online baselines with GroupDRO (e.g., ERM with GroupDRO at 84.95% average on R-A) remain below the proposed method.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use bi-level soft-label weighting with GroupDRO instead of vanilla soft-label vote weights | Toxicity classifier · Q-A | Average accuracy: 74.81 \(\pm\) 0.95 -> 78.41 \(\pm\) 0.24 | Helped |
| use bi-level soft-label weighting with GroupDRO instead of vanilla soft-label vote weights | Toxicity classifier · Q-A | Worst-group accuracy: 67.68 \(\pm\) 2.02 -> 69.44 \(\pm\) 0.13 | Helped |
| use bi-level soft-label weighting with GroupDRO instead of vanilla soft-label vote weights | Toxicity classifier · R-A | Average accuracy: 85.52 \(\pm\) 0.50 -> 89.80 \(\pm\) 0.61 | Helped |
| use bi-level soft-label weighting with GroupDRO instead of vanilla soft-label vote weights | Toxicity classifier · R-A | Worst-group accuracy: 62.57 \(\pm\) 4.42 -> 77.82 \(\pm\) 0.63 | Helped |
| replace GroupDRO with CVaR DRO (ablation) | Bi-level soft-label toxicity classifier · Q-A | Average accuracy: 78.41 \(\pm\) 0.24 -> 75.76 \(\pm\) 0.13 | Hurt |
| replace GroupDRO with CVaR DRO (ablation) | Bi-level soft-label toxicity classifier · Q-A | Worst-group accuracy: 69.44 \(\pm\) 0.13 -> 66.70 \(\pm\) 1.06 | Hurt |
| replace GroupDRO with CVaR DRO (ablation) | Bi-level soft-label toxicity classifier · R-A | Average accuracy: 89.80 \(\pm\) 0.61 -> 86.72 \(\pm\) 0.39 | Hurt |
| replace GroupDRO with CVaR DRO (ablation) | Bi-level soft-label toxicity classifier · R-A | Worst-group accuracy: 77.82 \(\pm\) 0.63 -> 68.30 \(\pm\) 0.13 | Hurt |
Surprising findings
- Average-label Learning and PRODEN collapse to roughly 19-36% average accuracy on Q-A and R-A, far below majority voting.
- CVaR DRO, a group-free alternative, underperforms GroupDRO in the paper's ablation.
Limitations
- Depends on quality and representativeness of annotations; no fairness audit of annotator bias.
- Requires a small clean validation set with ground truth and group annotations (topics and labels).
- Convergence theory relies on strong assumptions (lower-bounded gradient inner products, bounded gradients) and is not validated empirically against the theory.
Experimental setup
Datasets: Toxic question dataset (proprietary, from Sec3 Inc.) Toxic response dataset (proprietary, from Sec3 Inc.) HateXplain
Benchmarks: Toxic question classification (Q-A, Q-H, Q-L) Toxic response classification (R-A, R-H, R-L) HateXplain
Metrics: average accuracy worst-group accuracy
Baselines: Majority Voting PM Voting Snorkel Ensemble (post-training) Average-label Learning PRODEN Vanilla Soft Label CVaR DRO ERM with GroupDRO Consensus Only
Builds on: GroupDRO MAML-style bi-level optimization PRODEN Snorkel
Closest papers by meaning
- Learning from Noisy Labels via Conditional Distributionally Robust Optimization (NeurIPS 2024 · similarity 0.84)
- Unbiased Multi-Label Learning from Crowdsourced Annotations (ICML 2024 · similarity 0.83)
- Toxicity Detection for Free (NeurIPS 2024 · similarity 0.83)
- Like Oil and Water: Group Robustness Methods and Poisoning Defenses May Be at Odds (ICLR 2024 · similarity 0.82)
- Toward Robustness in Multi-Label Classification: A Data Augmentation Strategy against Imbalance and Noise (AAAI 2024 · similarity 0.82)
- BingoGuard: LLM Content Moderation Tools with Risk Levels (ICLR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.