Research map / Label-efficient and robust learning / Noisy labels
Retraining with Predicted Hard Labels Provably Increases Model Accuracy
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretically proves that retraining a linear classifier on its own predicted hard labels can beat the accuracy of training on noisy labels in a separable Gaussian-mixture setting, and shows consensus-based retraining improves label-DP training empirically.
Problem
Retraining a model with its own predicted hard labels often improves performance under noisy labels, but there is no theoretical characterization of when and why this helps; label DP training is a motivating application.
Approach
Analyze full retraining with predicted hard labels in a linearly separable Gaussian-type model with random label flips, showing the pseudo-labeler's error is sample-dependent and the retrained classifier's error decays with a much better exponent in (1-2p). Propose consensus-based retraining (keep only samples where predicted label equals given label) as a post-processing step for label DP that ad
Key results
- On CIFAR-100 with ResNet-18 at epsilon=3 label DP, consensus-based retraining reaches 29.98% vs 23.53% baseline (about 6.4 points gain).
- On AG News Subset with small BERT at epsilon=0.5, consensus-based RT reaches 80.95% vs 69.21% baseline (about 11.7 points gain).
- On CIFAR-10 with epsilon=1, consensus-based RT gives 63.84% vs 57.78% baseline and 60.07% for full RT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use full retraining with predicted hard labels instead of the initial noisy-label model | Linear classifier under random label flips · Linearly separable binary classification; stated sufficient conditions | population accuracy | Helped |
| Retrain on all predicted hard labels | ResNet-18 label-DP training · CIFAR-10 test, ε=1 | accuracy: 57.78 ± 1.13 -> 60.07 ± 0.63 | Helped |
| Retrain only on samples whose predicted and given labels match | ResNet-18 label-DP training · CIFAR-10 test, ε=1 | accuracy: 57.78 ± 1.13 -> 63.84 ± 0.56 | Helped |
| Retrain on all predicted hard labels | ResNet-18 label-DP training · CIFAR-100 test, ε=3 | accuracy: 23.53 ± 1.01 -> 24.42 ± 1.22 | Helped |
| Retrain only on samples whose predicted and given labels match | ResNet-18 label-DP training · CIFAR-100 test, ε=3 | accuracy: 23.53 ± 1.01 -> 29.98 ± 1.11 | Helped |
| Retrain only on samples whose predicted and given labels match | ResNet-50 linear probe · DomainNet test, ε=3 | accuracy: 23.60±0.92 -> 36.30±0.75 | Helped |
| Retrain only on samples whose predicted and given labels match | Small BERT label-DP training · AG News Subset test, ε=0.5 | accuracy: 69.21 ± 0.31 -> 80.95 ± 1.47 | Helped |
| Retrain on samples where predicted and given labels match after forward correction | ResNet-34 label-DP training · CIFAR-100 test, ε=4 | accuracy: 41.80 ± 0.90 -> 46.53 ± 0.76 | Helped |
| Retrain on samples where predicted and given labels match after symmetric CE initial training | ResNet-34 label-DP training · CIFAR-100 test, ε=5 | accuracy: 53.10±0.54 -> 56.13±0.25 | Helped |
| Increase model size from ResNet-18 to ResNet-34 | CIFAR-100 baseline label-DP training · CIFAR-100 test, ε=4 | accuracy: 44.53 ± 0.81 -> 37.53 ± 1.58 | Hurt |
| Retrain with predicted hard labels in the small-separation setting | Linear classifier under random label flips · Synthetic binary classification, γ² = 0.3 | test accuracy | No clear effect |
| Retrain with predicted hard labels in the large-separation setting | Linear classifier under random label flips · Synthetic binary classification, γ² = 0.5 | test accuracy: 89% -> 97.67% | Helped |
Surprising findings
- Full retraining with predicted hard labels gives only modest gains on CIFAR-100 at epsilon=3 (24.42% vs 23.53%) and yields no gain in the small-separation synthetic example, showing benefit depends on class separation.
- Consensus-based retraining with the same model outperforms full retraining and confidence-based retraining, even though it uses only about 11% of the training data at epsilon=3 on CIFAR-100.
Limitations
- Theory covers only full retraining with a simplified linear classifier (sum of label-weighted features) and uniform random label noise, not consensus-based retraining.
- Sample-size bounds have an upper limit on n that the authors say may be an artifact of the analysis.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 DomainNet AG News Subset
Benchmarks: CIFAR-10 CIFAR-100 DomainNet AG News Subset
Metrics: test accuracy population error bounds
Baselines: prior-based label DP method (two-stage) full retraining confidence-based retraining forward correction symmetric cross-entropy loss
Builds on: randomized response self-distillation self-training
Closest papers by meaning
- Enhancing Robustness of Last Layer Two-Stage Fair Model Corrections (NeurIPS 2024 · similarity 0.88)
- TrainRef: Curating Data with Label Distribution and Minimal Reference for Accurate Prediction and Reliable Confidence (ICLR 2026 · similarity 0.88)
- Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passing (NeurIPS 2025 · similarity 0.88)
- Enhancing Learning with Label Differential Privacy by Vector Approximation (ICLR 2025 · similarity 0.87)
- Learning from Noisy Labels via Self-Taught On-the-Fly Meta Loss Rescaling (AAAI 2025 · similarity 0.86)
- Optimal Transport Reweighting for Robust Learning under Spurious Correlations and Label Noise (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.