Research map / Label-efficient and robust learning / Recognition and long-tail
Low-Shot Validation: Active Importance Sampling for Estimating Classifier Performance on Rare Categories
Fait Poms, Vishnu Sarukkai, Ravi Teja Mullapudi, Nimit S. Sohoni, William R. Mark, Deva Ramanan, Kayvon Fatahalian
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add calibration (isotonic) to importance sampling (ablation) | SAWADE importance sampling · ImageNet50K | F1 MSE | Improved |
| add active importance sampling to calibration (ablation) | ISO / PLATT calibration-only · ImageNet50K | F1 MSE | Improved |
| average estimates over last iterations instead of last only (ablation) | ACIS-LAST · ImageNet50K | F1 MSE: fewer than 100 samples | Improved |
| use isotonic regression instead of Platt scaling (ablation) | ACIS · ImageNet50K | F1 MSE | Improved |
| calibration-only sampling at uniform random vs pure random (ablation) | ISO, PLATT vs RAND · ImageNet50K | F1 MSE: <100 samples | No clear effect |
| restrict sampling to adaptive top-K scored examples (ablation) | ACIS · rare categories | F1 MSE | Improved |
| average per-iteration variances instead of worst-case covariance 1 (ablation) | ACIS variance estimate · ImageNet1M | variance estimate accuracy | Improved |
| validate other models with sets curated for a different model | ACIS validation sets · ImageNet50K | F1 MSE: at most double the MSE of the model-specific estimator | Mixed |
| raise classifier threshold from 0.1-curated set to higher thresholds | ACIS validation set transfer · ImageNet50K | F1 MSE: thresholds 0.02 to 0.5 | Hurt |
Closest papers by meaning
- Enhanced Sample Selection with Confidence Tracking: Identifying Correctly Labeled Yet Hard-to-Learn Samples in Noisy Data (AAAI 2025 · similarity 0.86)
- Leveraging Uncertainty Estimates To Improve Classifier Performance (ICLR 2024 · similarity 0.86)
- What Does It Take to Build a Performant Selective Classifier? (NeurIPS 2025 · similarity 0.85)
- Provably Improving Generalization of Few-shot models with Synthetic Data (ICML 2025 · similarity 0.85)
- Bayesian Exploration of Pre-trained Models for Low-shot Image Classification (CVPR 2024 · similarity 0.85)
- Enhancing Semi-Supervised Learning via Representative and Diverse Sample Selection (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
