Research map / Label-efficient and robust learning / Recognition and long-tail
On the Connection between Pre-training Data Diversity and Fine-tuning Robustness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase pre-training data quantity (ablation) | ImageNet or iNaturalist pre-training · iWildCam-WILDS | effective robustness | Helped |
| pre-train with 25K natural images rather than train from scratch (ablation) | ImageNet or iNaturalist pre-training · iWildCam-WILDS | effective robustness | Helped |
| reduce natural-image pre-training data to 5K images (ablation) | ImageNet or iNaturalist pre-training · iWildCam-WILDS | effective robustness | No clear effect |
| use fine-grained rather than coarse pre-training labels (ablation) | ImageNet or iNaturalist pre-training · iWildCam-WILDS | effective robustness | Helped |
| pre-train on animal classes rather than object classes (ablation) | ImageNet pre-training · iWildCam-WILDS | effective robustness | No clear effect |
| trade more pre-training classes for more images per class, holding total samples fixed (ablation) | ImageNet pre-training · iWildCam-WILDS | effective robustness | No clear effect |
| reduce per-class subclass diversity by up to 3× while keeping data quantity and labels fixed (ablation) | ImageNet or iNaturalist pre-training · iWildCam-WILDS | effective robustness | No clear effect |
| use iNaturalist rather than an equal-sized ImageNet subset (ablation) | natural-image pre-training · iWildCam-WILDS | effective robustness | No clear effect |
| use FractalDB-1k synthetic images rather than natural images at the same data quantity (ablation) | pre-training · iWildCam-WILDS | effective robustness | Hurt |
| use natural-looking Stable Diffusion images rather than FractalDB-1k fractals (ablation) | synthetic pre-training · iWildCam-WILDS | effective robustness | Helped |
| increase Stable Diffusion pre-training data to 1.2M images rather than use natural data (ablation) | synthetic pre-training · iWildCam-WILDS | effective robustness | Hurt |
| vary LAION-5B pre-training size from 10M to 100M samples (ablation) | CLIP ViT-B/32 · ImageNet-derived distribution shifts | effective robustness | No clear effect |
Closest papers by meaning
- Using Pre-Training Can Improve Model Robustness and Uncertainty (ICML 2019 · similarity 0.90)
- Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks (ICLR 2024 · similarity 0.89)
- ImageNet Pre-training Also Transfers Non-robustness (AAAI 2023 · similarity 0.89)
- Improved Fine-Tuning by Better Leveraging Pre-Training Data (NeurIPS 2022 · similarity 0.89)
- Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuning (CVPR 2020 · similarity 0.89)
- Initialization Matters for Adversarial Transfer Learning (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.