Research map / Label-efficient and robust learning / Noisy labels
The Effect of Optimal Self-Distillation in Noisy Gaussian Mixture Model
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Uses the replica method to derive an exact asymptotic generalization-error formula for hyperparameter-tuned multi-stage self-distillation with a linear classifier on noisy Gaussian mixtures, and finds denoising via hard pseudo-labels drives gains, with early stopping and bias fixing as heuristics.
Problem
Self-distillation's effectiveness and optimal multi-stage hyperparameters are poorly understood, especially on noisy, non-separable data where pseudo-labels contain errors.
Approach
Model multi-stage SD with a linear (logistic or linear-MSE) classifier on noisy Gaussian mixtures in the proportional limit, derive the generalization error via a recursive replica method, and use the resulting low-dimensional formula to exhaustively search hyperparameters (regularization, inverse temperature) across stages.
Key results
- Hard pseudo-labels yield denoising gains comparable to soft labels; soft-label advantage is quantitatively small (under 0.4% in noiseless linear case).
- Optimal multi-stage SD benefits peak at moderate alpha; for alpha < Delta^2 the error at t to infinity converges to 0.5 (random guessing) under large regularization.
- On CIFAR-10 with ResNet-18 features, middle-alpha SD yields about 5% error reduction while large-alpha gains are near zero.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use hard rather than soft pseudo-labels (ablation) | Logistic 1-SD · Noisy Gaussian mixture | generalization error improvement | No clear effect |
| Use soft rather than hard pseudo-labels (ablation) | Logistic 1-SD · Some noise and dataset-size conditions | generalization error improvement | Helped |
| Use soft-label self-distillation instead of the base model | Linear 1-SD · Noiseless Gaussian mixture | generalization error improvement: under 0.4% | Helped |
| Add hard-pseudo-label self-distillation | Logistic 1-SD · Noisy Gaussian mixture | generalization error | Helped |
| Add multi-stage self-distillation | Linear t-SD · Gaussian mixture, intermediate dataset sizes | generalization error | Helped |
| Add multi-stage self-distillation | Linear t-SD · Gaussian mixture, small α | generalization error | No clear effect |
| Continue self-distillation through too many stages | Linear t-SD · Gaussian mixture, α<Δ², balanced labels | generalization error: 0.5 | Hurt |
| Stop self-distillation at an appropriate stage (ablation) | Linear t-SD · Gaussian mixture | generalization error | Helped |
| Fix the bias at its optimal 0-SD value during later stages (ablation) | Linear t-SD · Gaussian mixture, imbalanced labels | generalization error | Helped |
| Add 1-SD with soft pseudo-labels | Logistic 0-SD · CIFAR-10 cat vs. dog, pretrained ResNet-18 features, middle-α | generalization error improvement: ~ 5% performance gain | Helped |
| Add 1-SD with soft pseudo-labels | Logistic 0-SD · CIFAR-10 cat vs. dog, pretrained ResNet-18 features, large-α | generalization error | No clear effect |
| Use hard rather than soft pseudo-labels (ablation) | Logistic 1-SD · CIFAR-10 cat vs. dog, pretrained ResNet features | generalization error improvement | No clear effect |
Surprising findings
- Naively continuing multi-stage SD for many stages can drive generalization error to random guessing (0.5) when alpha < Delta^2.
- Dark knowledge from soft labels contributes only marginally compared with hard-label denoising.
Limitations
- Analysis limited to linear models on Gaussian mixture data; does not extend to deep networks.
- Replica method is not mathematically rigorous; results are conjectural analytically though numerically validated.
- Real-data experiments are a sanity check only (last-layer fine-tuning of pretrained ResNets on CIFAR-10 cat vs dog).
Experimental setup
Datasets: synthetic Gaussian mixture with label noise CIFAR-10 ImageNet1K_V2 pretraining for ResNet
Benchmarks: CIFAR-10 (cat vs dog, noisy labels)
Metrics: generalization error error improvement
Baselines: 0-SD base model noise-free 0-SD hard-label 1-SD
Builds on: knowledge distillation born again neural networks replica method Approximate Message Passing
Closest papers by meaning
- Rethinking Self-Distillation: Label Averaging and Enhanced Soft Label Refinement with Partial Labels (ICLR 2025 · similarity 0.88)
- Minimax Optimal Early-Stopped Gradient Descent for Gaussian Mixture Classification (NeurIPS 2026 · similarity 0.87)
- Boost Self-Supervised Dataset Distillation via Parameterization, Predefined Augmentation, and Approximation (ICLR 2025 · similarity 0.86)
- Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passing (NeurIPS 2025 · similarity 0.86)
- A Label is Worth A Thousand Images in Dataset Distillation (NeurIPS 2024 · similarity 0.85)
- Self-cognitive Denoising in the Presence of Multiple Noisy Label Sources (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.