Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
"A Framework for Efficient Model Evaluation through Stratification, Sampling, and Estimation"
Riccardo Fogliato, Pratik Patil, Mathew Monfort, Pietro Perona
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| stratify on model-performance predictions with k-means instead of simple random sampling | HT estimator, CLIP ViT-B/32 zero-shot · CIFAR-10 | MSE of accuracy estimate: about 10x gain in precision | Improved |
| stratified sampling with proportional allocation on tasks with uninformative proxy | HT estimator · DMLab Frames | MSE of accuracy estimate: no gain (ratio 0.98-1.01) | No clear effect |
| stratify on model predictions rather than image embeddings (ablation) | SSRS proportional + HT · CIFAR-10 | MSE relative to HT under SRS: 0.95 -> 0.74 | Improved |
| use a stronger surrogate model (CLIP ViT-L/14) to build the proxy (ablation) | SSRS proportional + HT · CIFAR-10 | MSE relative to HT under SRS: 0.74 -> 0.41 | Improved |
| calibrate the proxy with isotonic regression, Neyman allocation (ablation) | SSRS Neyman + HT · CIFAR-10 | MSE relative to HT under SRS: 0.35 -> 0.20 | Improved |
| calibrate the proxy for proportional allocation (ablation) | SSRS proportional + HT · Stanford Cars | MSE relative to HT under SRS: 0.34 -> 0.33 | No clear effect |
| use Neyman allocation with uncalibrated proxy instead of proportional allocation (ablation) | SSRS HT · DMLab Frames and PASCAL VOC 2007 | MSE relative to HT under SRS: 1.00 -> 1.09 (f, DMLab); 0.75 -> 1.07 (f, VOC) | Hurt |
| use Neyman allocation with calibrated surrogate proxy (ablation) | SSRS HT · Stanford Cars | MSE relative to HT under SRS: 0.33 -> 0.23 | Improved |
| use difference estimator under SRS with uncalibrated proxy | SRS + DF · 35 CLIP-Benchmark tasks | MSE relative to HT under SRS: e.g. 0.45 on CIFAR-10 vs 1.82 on dSprites (y position) | Mixed |
| calibrate the proxy used by the difference estimator (ablation) | SRS + DF · 35 CLIP-Benchmark tasks | MSE relative to HT under SRS: 1.82 -> 0.99 (dSprites y position) | Improved |
| use difference estimator on top of stratified proportional sampling | SSRS proportional · 35 CLIP-Benchmark tasks | MSE | No clear effect |
| use linear probing instead of zero-shot predictions in the evaluated CLIP model (ablation) | efficiency of all methods · CLIP linear probing | relative efficiency | Improved |
Closest papers by meaning
- Are Labels Always Necessary for Classifier Accuracy Evaluation? (CVPR 2021 · similarity 0.85)
- AutoEval: Are Labels Always Necessary for Classifier Accuracy Evaluation? (TPAMI 2021 · similarity 0.85)
- Bayesian Exploration of Pre-trained Models for Low-shot Image Classification (CVPR 2024 · similarity 0.84)
- Evaluating Model Bias Requires Characterizing its Mistakes (ICML 2024 · similarity 0.84)
- Benchmarking the Attribution Quality of Vision Models (NeurIPS 2024 · similarity 0.84)
- Divide-and-Aggregate Learning for Evaluating Performance on Unlabeled Data (AAAI 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
