Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Benchmarking Low-Shot Robustness to Natural Distribution Shifts
Aaditya Singh, Kartik Sarangmath, Prithvijit Chattopadhyay, Judy Hoffman
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use self-supervised ViTs instead of SSL CNNs and supervised models (ablation) | ImageNet pre-trained low-shot models · ImageNet, iWildCam, Camelyon | ID/OOD accuracy | Improved |
| choose MSN ViTB-16 vs DINO / ViTS-16 initialization (ablation) | low-shot fine-tuning · ImageNet, iWildCam, Camelyon | OOD accuracy: ImageNet OOD 22.81 (MSN ViTB) vs iWildCam 19.99 (MSN ViTS) vs Camelyon 85.09 (DINO ViTS) | Mixed |
| use CLIP instead of ImageNet-1k pre-trained DINO ViTB-16 (ablation) | low-shot ViTB-16 · iWildCam OOD | avg per-class accuracy: 19.82 -> 19.10 | Hurt |
| use CLIP instead of ImageNet-1k pre-trained DINO ViTB-16 (ablation) | low-shot ViTB-16 · Camelyon OOD | avg per-class accuracy: 84.33 -> 77.3 | Hurt |
| use CLIP zero-shot instead of low-shot trained models (ablation) | ImageNet · ImageNet OOD | top-1 accuracy: 27.50 -> 57.37 | Improved |
| use IN21k supervised ViT instead of IN1k supervised ViT (ablation) | iWildCam, Camelyon · iWildCam OOD | avg per-class accuracy: 18.57 -> 16.90 | Hurt |
| add WiSE-FT on CLIP | CLIP ViTB-16 · ImageNet, iWildCam, Camelyon (all regimes) | effective and relative robustness | Improved |
| add robustness interventions (LP-FT, WiSE-FT, Soups) on MSN ViTB-16 | MSN ViTB-16 · iWildCam | effective/relative robustness | No clear effect |
| add robustness interventions on MSN/CLIP ViTB-16 in low-shot regimes | ViTB-16 · Camelyon extreme and moderate low-shot | relative robustness | Hurt |
| add RobustViT and other interventions | MSN ViTB-16 · ImageNet | effective and relative robustness | Improved |
| set WiSE-FT alpha to 1 instead of 0.5 with CLIP (ablation) | WiSE-FT + CLIP · Camelyon full-shot OOD | avg per-class accuracy: 70.60 -> 94.27 | Improved |
| use Baseline++ instead of logistic regression classifier (ablation) | MSN ViTS-16 · Camelyon low-shot OOD | avg per-class accuracy: 69.73 -> 75.66 | Improved |
Closest papers by meaning
- Measuring Robustness to Natural Distribution Shifts in Image Classification (NeurIPS 2020 · similarity 0.92)
- BREEDS: Benchmarks for Subpopulation Shift (ICLR 2021 · similarity 0.90)
- The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization (ICCV 2021 · similarity 0.89)
- Robustness of Probabilistic Models to Low-Quality Data: A Multi-Perspective Analysis (ICLR 2026 · similarity 0.89)
- Towards Robustness to Natural Variations and Distribution Shift (Student Abstract) (AAAI 2024 · similarity 0.88)
- Dual Risk Minimization: Towards Next-Level Robustness in Fine-tuning Zero-Shot Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
