AI Paper Atlas

Research map / Learning theory and algorithms / Sample complexity

Evaluating Bayes Error Estimators on Real-World Datasets with FeeBee

Cedric Renggli, Luka Rimanic, Nora Hollenstein, Ce Zhang

NeurIPS 2021 · Learning theory and algorithms › Sample complexity · approach: Optimal mean estimation and subsampling · 1 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use kNN or kNN-LOO rather than other estimators for the lower-bound estimateBER estimators · MNIST, CIFAR10, CIFAR100, IMDB, SST2L_D(m): MNIST: 0.02; CIFAR10: 0.03; CIFAR100: 0.06; IMDB: 0.15; SST2: 0.29Improved
Use kNN-LOO instead of other tested estimatorsBER estimators · IMDBL_D(m): 0.15Improved
Choose kNN-Extrapolate as the upper-bound estimatorBER estimators · CIFAR100, IMDBU_D(m): CIFAR100: 0.21; IMDB: 0.28Improved
Use logistic regression rather than 1NN for the upper-bound estimateBER estimators · YELPU_D(m): 1NN: 0.38; LR Model (0.8): 0.10Improved
Increase k from 1 to 5, 7, or 9 (ablation)kNN · CIFAR10L_D(m): 1NN: 0.05; 5NN: 0.11; 7NN: 0.14; 9NN: 0.16Hurt
Increase k from 1 to 3 (ablation)kNN · CIFAR10L_D(m): 1NN: 0.05; 3NN: 0.04Improved
Increase k from 1 to 3 (ablation)kNN · IMDBL_D(m): 1NN: 0.26; 3NN: 0.30Hurt
Increase the LR scaling from 0.8 to 0.95 (ablation)Logistic regression BER estimator · YELPL_D(m): 0.08 -> 0.06Improved
Increase the LR scaling from 0.8 to 0.95 (ablation)Logistic regression BER estimator · YELPU_D(m): 0.10 -> 0.10No clear effect
Use raw pixel features rather than selecting the best feature transformation (ablation)kNN · CIFAR10L_D(m): 0.03 -> 0.39Hurt
Select hyperparameters and transformation by minimizing ell_D,m(0) instead of minimizing L_D(m) (ablation)DE-kNN, KDE, 1NN-kNN, kNN-Extrapolate · MNIST, CIFAR10, CIFAR100, SST2L_D(m): MNIST: 0.24, 0.59, 0.56, 0.37; CIFAR10: 0.21, 0.64, 0.52, 0.42; CIFAR100: 0.10, 0.69, 0.30, 0.59; SST2: 0.00, 0.50, 0.20Hurt
Select hyperparameters and transformation by minimizing ell_D,m(0) (ablation)1NN and GHP · MNIST, CIFAR10, CIFAR100L_D(m): 0.00No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy