Research map / Label-efficient and robust learning / Noisy labels
Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies how synthetic label noise in supervised/CLIP pre-training affects ID vs OOD transfer, finds slight noise helps ID but hurts OOD, and proposes NMTune, a light-weight black-box MLP tuning with covariance, MSE and singular-value regularizers that improves frozen pre-trained vision and language models.
Problem
Label noise in large-scale pre-training data is hard to remove and users often only have black-box access to pre-trained models, so its effect on downstream ID and OOD generalization is unclear and unmitigated.
Approach
Shows noise in pre-training reshapes the feature singular-value spectrum (SVE up, largest singular value ratio down), and proposes NMTune, which trains a 2-layer MLP on frozen features with consistency, covariance, and dominant-singular-value regularizers to rectify these effects.
Key results
- On vision models, NMTune improves ID average accuracy over LP for all five pre-trained models, e.g., Swin-L 81.91 to 84.16 and ConvNext-L 88.86 to 89.48.
- On OOD tasks, NMTune raises average accuracy, e.g., ConvNext-L 66.86 to 70.30 and Swin-L 50.88 to 52.35 on DomainNet.
- On language models, NMTune improves GLUE-X OOD average for text-ada-002 from 44.06 (LP) to 53.48 and GPT-2 from 36.68 to 39.07.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add NMTune regularization to MLP tuning (ablation) | Vision models EfficientNet-B3, ResNetv2-152x2, Swin-L, ConvNext-L, and ViT-L · 14 vision ID tasks | average accuracy: 76.87 -> 77.63; 77.58 -> 78.43; 82.51 -> 84.16; 88.53 -> 89.48; 87.23 -> 88.57 | Helped |
| Add NMTune regularization to MLP tuning (ablation) | Vision models EfficientNet-B3, ResNetv2-152x2, Swin-L, ConvNext-L, and ViT-L · 4 vision OOD tasks | average accuracy: 45.95 -> 46.84; 41.73 -> 42.42; 51.21 -> 52.35; 68.43 -> 70.30; 69.50 -> 70.47 | Helped |
| Add NMTune regularization to MLP tuning (ablation) | BERT-L, RoBERTa-L, GPT-2, and text-ada-002 · GLUE | average score: 69.78 -> 70.26; 70.27 -> 70.97; 57.19 -> 58.09; 63.89 -> 65.99 | Helped |
| Add NMTune regularization to MLP tuning (ablation) | BERT-L, RoBERTa-L, GPT-2, and text-ada-002 · GLUE-X | average score: 50.62 -> 51.63; 45.22 -> 47.01; 37.24 -> 39.07; 51.31 -> 53.48 | Helped |
| Add slight label noise (5% or 10%) during supervised pre-training | ResNet-50 pre-trained on ImageNet-1K or YFCC15M · In-domain downstream tasks | accuracy | Helped |
| Add label noise during supervised pre-training | ResNet-50 pre-trained on ImageNet-1K or YFCC15M · Out-of-domain downstream tasks | robustness performance | Hurt |
| Increase pre-training label noise beyond 5% or 10% | ResNet-50 pre-trained on ImageNet-1K or YFCC15M · In-domain downstream tasks | accuracy | Hurt |
| Use MLP tuning without NMTune regularization instead of linear probing (ablation) | Noisy pre-trained models · In-domain and out-of-domain downstream tasks | generalization performance | Mixed |
Surprising findings
- Slight label noise (5-10%) in pre-training improves in-domain downstream performance while always hurting OOD robustness.
- Adding an MLP alone on frozen features can reduce in-domain performance relative to LP for some models, suggesting extra parameters do not mitigate noise.
Limitations
- Analysis uses only ResNet-50 on small-scale pre-training due to limited compute.
- Results are not comparable to white-box full fine-tuning, since feature extractors are frozen.
- Noise is synthetic (symmetric label flips and random text swaps), not real-world noise.
Experimental setup
Datasets: ImageNet-1K YFCC15M CIFAR-10 CIFAR-100 Flowers102 Food101 OxfordPet StanfordCars FGVCAircraft SVHN DTD Caltech101
Benchmarks: CIFAR-10 CIFAR-100 Flowers102 Food101 OxfordPet StanfordCars FGVCAircraft SVHN DTD Caltech101 EuroSAT PatchCamelyon
Metrics: accuracy F1 score singular value entropy largest singular value ratio
Baselines: linear probing MLP tuning
Builds on: linear probing Barlow Twins-style covariance regularization CLIP GLUE-X
Closest papers by meaning
- On the Connection between Pre-training Data Diversity and Fine-tuning Robustness (NeurIPS 2023 · similarity 0.89)
- On the Role of Label Noise in the Feature Learning Process (ICML 2025 · similarity 0.89)
- How Does Label Noise Gradient Descent Improve Generalization in the Low SNR Regime? (NeurIPS 2025 · similarity 0.88)
- An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence (NeurIPS 2026 · similarity 0.87)
- Handling Label Noise via Instance-Level Difficulty Modeling and Dynamic Optimization (NeurIPS 2025 · similarity 0.87)
- Mitigating Label Noise through Data Ambiguation (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.