AI Paper Atlas

Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes

"A Framework for Efficient Model Evaluation through Stratification, Sampling, and Estimation"

Riccardo Fogliato, Pratik Patil, Mathew Monfort, Pietro Perona

ECCV 2024 · Probabilistic and scientific modeling › Uncertainty and Gaussian processes · approach: Priors and functional variational uncertainty · 7 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
stratify on model-performance predictions with k-means instead of simple random samplingHT estimator, CLIP ViT-B/32 zero-shot · CIFAR-10MSE of accuracy estimate: about 10x gain in precisionImproved
stratified sampling with proportional allocation on tasks with uninformative proxyHT estimator · DMLab FramesMSE of accuracy estimate: no gain (ratio 0.98-1.01)No clear effect
stratify on model predictions rather than image embeddings (ablation)SSRS proportional + HT · CIFAR-10MSE relative to HT under SRS: 0.95 -> 0.74Improved
use a stronger surrogate model (CLIP ViT-L/14) to build the proxy (ablation)SSRS proportional + HT · CIFAR-10MSE relative to HT under SRS: 0.74 -> 0.41Improved
calibrate the proxy with isotonic regression, Neyman allocation (ablation)SSRS Neyman + HT · CIFAR-10MSE relative to HT under SRS: 0.35 -> 0.20Improved
calibrate the proxy for proportional allocation (ablation)SSRS proportional + HT · Stanford CarsMSE relative to HT under SRS: 0.34 -> 0.33No clear effect
use Neyman allocation with uncalibrated proxy instead of proportional allocation (ablation)SSRS HT · DMLab Frames and PASCAL VOC 2007MSE relative to HT under SRS: 1.00 -> 1.09 (f, DMLab); 0.75 -> 1.07 (f, VOC)Hurt
use Neyman allocation with calibrated surrogate proxy (ablation)SSRS HT · Stanford CarsMSE relative to HT under SRS: 0.33 -> 0.23Improved
use difference estimator under SRS with uncalibrated proxySRS + DF · 35 CLIP-Benchmark tasksMSE relative to HT under SRS: e.g. 0.45 on CIFAR-10 vs 1.82 on dSprites (y position)Mixed
calibrate the proxy used by the difference estimator (ablation)SRS + DF · 35 CLIP-Benchmark tasksMSE relative to HT under SRS: 1.82 -> 0.99 (dSprites y position)Improved
use difference estimator on top of stratified proportional samplingSSRS proportional · 35 CLIP-Benchmark tasksMSENo clear effect
use linear probing instead of zero-shot predictions in the evaluated CLIP model (ablation)efficiency of all methods · CLIP linear probingrelative efficiencyImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy