Research map / Learning theory and algorithms / Sample complexity
Evaluating Bayes Error Estimators on Real-World Datasets with FeeBee
Cedric Renggli, Luka Rimanic, Nora Hollenstein, Ce Zhang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use kNN or kNN-LOO rather than other estimators for the lower-bound estimate | BER estimators · MNIST, CIFAR10, CIFAR100, IMDB, SST2 | L_D(m): MNIST: 0.02; CIFAR10: 0.03; CIFAR100: 0.06; IMDB: 0.15; SST2: 0.29 | Improved |
| Use kNN-LOO instead of other tested estimators | BER estimators · IMDB | L_D(m): 0.15 | Improved |
| Choose kNN-Extrapolate as the upper-bound estimator | BER estimators · CIFAR100, IMDB | U_D(m): CIFAR100: 0.21; IMDB: 0.28 | Improved |
| Use logistic regression rather than 1NN for the upper-bound estimate | BER estimators · YELP | U_D(m): 1NN: 0.38; LR Model (0.8): 0.10 | Improved |
| Increase k from 1 to 5, 7, or 9 (ablation) | kNN · CIFAR10 | L_D(m): 1NN: 0.05; 5NN: 0.11; 7NN: 0.14; 9NN: 0.16 | Hurt |
| Increase k from 1 to 3 (ablation) | kNN · CIFAR10 | L_D(m): 1NN: 0.05; 3NN: 0.04 | Improved |
| Increase k from 1 to 3 (ablation) | kNN · IMDB | L_D(m): 1NN: 0.26; 3NN: 0.30 | Hurt |
| Increase the LR scaling from 0.8 to 0.95 (ablation) | Logistic regression BER estimator · YELP | L_D(m): 0.08 -> 0.06 | Improved |
| Increase the LR scaling from 0.8 to 0.95 (ablation) | Logistic regression BER estimator · YELP | U_D(m): 0.10 -> 0.10 | No clear effect |
| Use raw pixel features rather than selecting the best feature transformation (ablation) | kNN · CIFAR10 | L_D(m): 0.03 -> 0.39 | Hurt |
| Select hyperparameters and transformation by minimizing ell_D,m(0) instead of minimizing L_D(m) (ablation) | DE-kNN, KDE, 1NN-kNN, kNN-Extrapolate · MNIST, CIFAR10, CIFAR100, SST2 | L_D(m): MNIST: 0.24, 0.59, 0.56, 0.37; CIFAR10: 0.21, 0.64, 0.52, 0.42; CIFAR100: 0.10, 0.69, 0.30, 0.59; SST2: 0.00, 0.50, 0.20 | Hurt |
| Select hyperparameters and transformation by minimizing ell_D,m(0) (ablation) | 1NN and GHP · MNIST, CIFAR10, CIFAR100 | L_D(m): 0.00 | No clear effect |
Closest papers by meaning
- Bayes-Optimal BER and AUC: Estimation and Evaluation of Estimators (NeurIPS 2026 · similarity 0.91)
- Evaluating Classification Model Against Bayes Error Rate (TPAMI 2023 · similarity 0.89)
- SimPro: A Simple Probabilistic Framework Towards Realistic Long-Tailed Semi-Supervised Learning (ICML 2024 · similarity 0.85)
- What Does It Take to Build a Performant Selective Classifier? (NeurIPS 2025 · similarity 0.85)
- Leveraging Uncertainty Estimates To Improve Classifier Performance (ICLR 2024 · similarity 0.84)
- Recursive PAC-Bayes: A Frequentist Approach to Sequential Prior Updates with No Information Loss (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
