Research map / Label-efficient and robust learning / Recognition and long-tail
Impact of Noisy Supervision in Foundation Model Learning
Hao Chen, Zihan Wang, Ran Tao, Hongxin Wei, Xing Xie, Masashi Sugiyama, Bhiksha Raj, Jindong Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add NMTune regularization to linear probing | ViT-L (Laion-2B CLIP) · 14 ID vision datasets | accuracy: 86.85 -> 88.57 | Improved |
| add NMTune regularization to linear probing | ConvNext-L (Laion-2B CLIP) · 4 OOD vision datasets | accuracy: 66.86 -> 70.30 | Improved |
| add regularization terms to MLP tuning (ablation) | ViT-L · 4 OOD vision datasets | accuracy: 69.50 -> 70.47 | Improved |
| add an extra MLP head alone (no regularization) to LP (ablation) | ConvNext-L · 14 ID vision datasets | accuracy: 88.86 -> 88.53 | Hurt |
| add NMTune to API embedding model | text-ada-002 · GLUE-X (OOD) | score: 44.06 -> 53.48 | Improved |
| add NMTune to MLP tuning on GLUE (ablation) | BERT-L · GLUE-X (OOD) | score: 50.62 -> 51.63 | Improved |
| raise pre-training label noise from 0% to 5% | ResNet-50 / ViT-B-16 (IN-1K FS, CLIP) · downstream ID classification (LP) | accuracy | Improved |
| raise pre-training label noise from 0% to 5% | ResNet-50 / ViT-B-16 · downstream OOD classification (LP, LoRA) | accuracy | Hurt |
| raise pre-training noise from 0% to 30% | ViT-B-16 IN-1K FS · IN-1K validation | accuracy: 78.70 -> 68.12 | Hurt |
| raise pre-training noise from 0% to 5% | Faster R-CNN with ResNet-50 · COCO 2017 detection | AP box: 38.5 -> 38.6 | No clear effect |
| raise pre-training noise from 0% to 30% | Mask R-CNN with ResNet-50 · COCO 2017 segmentation | AP mask: 31.3 -> 30.3 | Hurt |
| tune more parameters (LP to LoRA to full fine-tuning) | noisy pre-trained ViT-B-16 · ID and OOD classification | accuracy | Improved |
Closest papers by meaning
- Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks (ICLR 2024 · similarity 0.96)
- An Empirical Study on Noisy Data and LLM Pretraining Loss Divergence (NeurIPS 2026 · similarity 0.88)
- Handling Label Noise via Instance-Level Difficulty Modeling and Dynamic Optimization (NeurIPS 2025 · similarity 0.87)
- On the Role of Label Noise in the Feature Learning Process (ICML 2025 · similarity 0.87)
- Specialized Foundation Models Struggle to Beat Supervised Baselines (ICLR 2025 · similarity 0.87)
- Quick-Tune: Quickly Learning Which Pretrained Model to Finetune and How (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
