Research map / Learning theory and algorithms / Prediction and decision losses
Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passing
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives the Bayes-optimal aggregator for combining a model's predictions with noisy labels during iterative retraining, using approximate message passing analysis for GMM and GLM binary classification, and applies a practical version (BayesMix RT) to linear probing of ResNet-50.
Problem
How to optimally combine a model's own predicted labels with given noisy labels when retraining a classifier over multiple rounds in binary classification.
Approach
Uses AMP state evolution for iterative retraining under GMM and GLM data models to derive the aggregator g_t that maximizes signal-to-noise of the next iterate (Bayes-optimal given the prediction and noisy label), and proves that retraining helps only when the initial model is poor (monotone state evolution with fixed points).
Key results
- On MedMNIST Pneumonia at p=0.45, BayesMix RT reaches 71.42% after 10 iterations vs 68.06% (full RT) and 70.03% (consensus RT), though consensus RT is best after 1 iteration.
- On Food-101 pho vs ramen at p=0.45, BayesMix RT reaches 76.60% after 10 iterations vs 62.60% (full RT) and 64.87% (consensus RT).
- Synthetic GMM experiments show state evolution closely matches AMP simulations, and the optimal aggregator outperforms full and consensus retraining at every iteration.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BayesMix RT instead of consensus-based RT | Linear probing with ResNet-50 pretrained on ImageNet · MedMNIST Pneumonia, p = 0.45 | test accuracy: 68.32 ± 1.88 -> 65.06 ± 2.55 (iteration 1); 70.03 ± 0.73 -> 71.42 ± 2.43 (iteration 10) | Mixed |
| Use BayesMix RT instead of consensus-based RT | Linear probing with ResNet-50 pretrained on ImageNet · Food-101 Pho vs Ramen, p = 0.45 | test accuracy: 63.60 ± 3.22 -> 60.53 ± 3.27 (iteration 1); 64.87 ± 4.07 -> 76.60 ± 6.00 (iteration 10) | Mixed |
| Retrain for 10 iterations rather than 1 with BayesMix RT | Linear probing with ResNet-50 pretrained on ImageNet · MedMNIST Pneumonia, p = 0.45 | test accuracy: 65.06 ± 2.55 -> 71.42 ± 2.43 | Helped |
| Retrain for 10 iterations rather than 1 with BayesMix RT | Linear probing with ResNet-50 pretrained on ImageNet · Food-101 Pho vs Ramen, p = 0.45 | test accuracy: 60.53 ± 3.27 -> 76.60 ± 6.00 | Helped |
Surprising findings
- Retraining can hurt performance: if the initial model is already good (eta_1 above the smallest fixed point), state evolution decreases and error increases over rounds.
- Consensus-based RT is best after one retraining iteration on Pneumonia and Food-101, yet the Bayes-optimal method wins after 10 iterations.
Limitations
- Theory restricted to linear models with binary classification and uniform label noise.
- Practical method for deep linear probing relies on a bimodal GMM fit to logits and is a heuristic adaptation rather than the exact derived aggregator.
- Full and consensus retraining were simulated without memory correction terms because their aggregators are not Lipschitz.
Experimental setup
Datasets: MedMNIST Pneumonia Food-101 (pho vs ramen)
Benchmarks: MedMNIST Pneumonia Food-101 pho vs ramen
Metrics: test accuracy classification error (state evolution prediction)
Baselines: full retraining consensus-based retraining vanilla linear probing
Builds on: approximate message passing generalized AMP self-distillation full retraining consensus-based retraining
Closest papers by meaning
- Retraining with Predicted Hard Labels Provably Increases Model Accuracy (ICML 2025 · similarity 0.88)
- The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Model (NeurIPS 2025 · similarity 0.86)
- Improving Generalization via Meta-Learning on Hard Samples (CVPR 2024 · similarity 0.85)
- Minimax Optimal Early-Stopped Gradient Descent for Gaussian Mixture Classification (NeurIPS 2026 · similarity 0.85)
- Optimal Transport Reweighting for Robust Learning under Spurious Correlations and Label Noise (NeurIPS 2026 · similarity 0.85)
- When to retrain a machine learning model (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.