AI Paper Atlas

Research map / Learning theory and algorithms / Sample complexity

Evaluating model performance under worst-case subpopulations

Mike Li, Hongseok Namkoong, Shangzhou Xia

NeurIPS 2021 · Learning theory and algorithms › Sample complexity · approach: Optimal mean estimation and subsampling

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
fine-tune zero-shot CLIP on FMoWCLIP · FMoW ID valaccuracy: 19.3% -> 70.2%Improved
fine-tune zero-shot CLIP on FMoWCLIP · FMoW OOD robustnessrelative robustnessHurt
weight-space ensemble of zero-shot and fine-tuned CLIP (WiSE-FT, lambda=0.4)CLIP fine-tuned · FMoW OOD testaccuracy: 0.53 -> 0.56Improved
CLIP WiSE-FT versus DenseNet ERMFMoW models · FMoW OOD Africacross-entropy loss: 5.41 -> 3.08Improved
CLIP WiSE-FT versus DenseNet ERMFMoW models · FMoW OOD Africaaccuracy: 0.33 -> 0.38Improved
CLIP WiSE-FT versus DenseNet ERMFMoW models · FMoW OOD testcross-entropy loss: 3.50 -> 2.84Improved
DenseNet IRM versus DenseNet ERMDenseNet · FMoW ID valcross-entropy loss: 2.78 -> 2.44Improved
DenseNet IRM versus DenseNet ERMDenseNet · FMoW OOD testaccuracy: 0.53 -> 0.51Hurt
DPN-68 versus DenseNet ERMFMoW models · FMoW OOD Africacross-entropy loss: 5.41 -> 5.61Hurt
include label Y in subpopulation definition Zworst-case estimator W_alpha · FMoW ID valworst-case lossHurt
reduce subpopulation size alphaImageNet pretrained models vs CLIP WiSE-FT · FMoW ID valworst-case loss gapHurt
evaluate linear regression vs expressive models on worst-case subpopulationslinear model · Warfarin dosingsquared errorNo clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy