Research map / Label-efficient and robust learning / Noisy labels
Learning from Noisy Labels via Self-Taught On-the-Fly Meta Loss Rescaling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STORM, an unsupervised on-the-fly meta loss rescaling method that learns a per-sample loss weighting function from model features while meta-validating on the noisy training data itself, evaluated on noisy NLP classification and MultiWOZ dialogue state tracking.
Problem
Learning from noisy labels in NLP without clean seed or validation data, where on-the-fly loss reweighting usually requires clean data and sample selection causes information loss.
Approach
Learn a rescaling network from model-derived features (losses, probabilities, KL/OVL divergences, agreement) via meta learning, where the outer-loop validation batches are sampled from the noisy training set and also rescaled by the same rescaler, so no clean data is needed.
Key results
- Under 30% uniform noise, STORM averages 74.7 across six settings versus 70.1 for no rescaling and 73.3 for meta learning with clean validation.
- On SMS with uniform noise, STORM improves F1 by roughly 19-21 points absolute over no rescaling.
- On MultiWOZ with real noise, STORM improves JGA from 65.0 to 67.1 (about 2 points) with TripPy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply STORM loss rescaling | RoBERTa-base · SMS, 30% uniform noise | F1: 63.6$\pm$4.5 -> 82.3$\pm$3.1 | Helped |
| apply STORM loss rescaling | RoBERTa-base · MultiWOZ, real noise | JGA: 65.0$\pm$0.8 -> 67.1$\pm$0.7 | Helped |
| apply STORM loss rescaling | RoBERTa-base · YouTube, 30% uniform noise | accuracy: 89.5$\pm$1.5 -> 91.0$\pm$0.7$^{\uparrow}$ | Helped |
| apply STORM loss rescaling | RoBERTa-base · CoLA, 30% uniform noise | Matthews correlation: 49.8$\pm$2.5 -> 51.8$\pm$2.1$^{\uparrow}$ | Helped |
| apply STORM loss rescaling | RoBERTa-base · MRPC, 30% uniform noise | F1: 84.6$\pm$2.1 -> 86.8$\pm$1.2$^{\Uparrow}$ | Helped |
| apply STORM loss rescaling | RoBERTa-base · RTE, 30% uniform noise | F1: 67.9$\pm$1.6 -> 68.9$\pm$2.7 | No clear effect |
| use binary rather than continuous rescaling (ablation) | STORM · SMS and CoLA, 30% uniform noise | F1 / Matthews correlation: SMS: 82.3$\pm$3.1 -> 15.3$\pm$24.7$^{\Downarrow}$; CoLA: 51.8$\pm$2.1 -> 0.0$\pm$0.0$^{\Downarrow}$ | Hurt |
| use 2 inner loops instead of 1 (ablation) | STORM · MultiWOZ, real noise | JGA: 67.1$\pm$0.7 -> 61.1$\pm$6.8 | Hurt |
| use 10 forward passes instead of 3 (ablation) | STORM · MultiWOZ, real noise | JGA: 67.1$\pm$0.7 -> 62.9$\pm$2.7 | Hurt |
| remove class-dependent rescaling (ablation) | STORM · MultiWOZ, real noise | JGA: 67.1$\pm$0.7 -> 64.5$\pm$1.5 | Hurt |
| remove extra rescaler features (ablation) | STORM · MultiWOZ, real noise | JGA: 67.1$\pm$0.7 -> 39.2$\pm$18.6$^{\downarrow}$ | Hurt |
| remove meta learning (ablation) | STORM · MultiWOZ, real noise | JGA: 67.1$\pm$0.7 -> 65.0$\pm$0.9 | No clear effect |
Surprising findings
- Using a clean validation set for the meta update did not generally improve rescaling quality or performance, and was detrimental in the outer loop.
- Binary (hard filtering) rescaling collapsed on some tasks, e.g., 0.0 on CoLA and 15.3 F1 on SMS under 30% noise.
Limitations
- Confirmation bias risk when noisy labels are consistently wrong for a class or for underrepresented classes (e.g., dontcare in MultiWOZ).
- Potentially unsuitable for extremely noisy data where clean samples are outnumbered.
- Gains on clean data are not significant; results on real noise are modest (about 2 JGA points).
Experimental setup
Datasets: YouTube spam SMS spam CoLA MRPC RTE MultiWOZ 2.1 MultiWOZ 2.4
Benchmarks: YouTube spam SMS spam CoLA MRPC RTE MultiWOZ 2.4
Metrics: accuracy F1 Matthews correlation joint goal accuracy calibration error
Baselines: No rescaling AGRA Meta learning w/ clean val. STORM w/ binary rescaling
Builds on: AGRA TripPy RoBERTa learning to reweight examples
Closest papers by meaning
- Dynamic Loss-Based Sample Reweighting for Improved Large Language Model Pretraining (ICLR 2025 · similarity 0.88)
- Improving Generalization via Meta-Learning on Hard Samples (CVPR 2024 · similarity 0.87)
- Self Iterative Label Refinement via Robust Unlabeled Learning (NeurIPS 2025 · similarity 0.87)
- Retraining with Predicted Hard Labels Provably Increases Model Accuracy (ICML 2025 · similarity 0.86)
- Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning (ICML 2025 · similarity 0.86)
- Regroup Median Loss for Combating Label Noise (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.