Research map / Label-efficient and robust learning / Recognition and long-tail
Is the Performance of My Deep Network Too Good to Be True? A Direct Approach to Estimating the Bayes Error in Binary Classification
Takashi Ishida, Ikko Yamane, Nontawat Charoenphakdee, Gang Niu, Masashi Sugiyama
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Estimate Bayes error directly by averaging uncertainty labels instead of training a model | Binary classification with clean soft or uncertainty labels · Synthetic binary classification | Bayes error estimation accuracy | Improved |
| Train an MLP instead of directly averaging uncertainty labels | Binary classification · Synthetic binary classification | Bayes error estimation accuracy | Hurt |
| Use the original minimum-based estimator with noisy soft labels rather than clean labels (ablation) | Proposed Bayes error estimator · Synthetic binary classification with noisy labels | Bayes error estimation accuracy | Hurt |
| Add sign labels and use the noise-corrected estimator (ablation) | Proposed Bayes error estimator · Synthetic binary classification with noisy labels | Bayes error estimation accuracy | Improved |
| Use positive-confidence data instead of positive-negative data (ablation) | Proposed Bayes error estimator · Synthetic binary classification | Bayes error estimation accuracy | Mixed |
| Use the classifier's probabilistic output in the Bayes-error equation instead of directly averaging soft labels | Classifier-BER · Synthetic binary classification | Bayes error estimation accuracy | Hurt |
| Use hard-label classifier probabilities rather than soft-label classifier probabilities in classifier-BER (ablation) | Classifier-BER · Synthetic binary classification | Bayes error estimation accuracy | Hurt |
| Compare ResNet-18 test error with the estimated Bayes error | ResNet-18 · Fashion-MNIST-H | error: 3.852% (± 0.041%) vs 3.478% (± 0.079%) | Mixed |
Closest papers by meaning
- Evaluating State-of-the-Art Classification Models Against Bayes Optimality (NeurIPS 2021 · similarity 0.88)
- Demystifying the Optimal Performance of Multi-Class Classification (NeurIPS 2023 · similarity 0.87)
- How Good is the Bayes Posterior in Deep Neural Networks Really? (ICML 2020 · similarity 0.87)
- Bayes-optimal Learning of Deep Random Networks of Extensive-width (ICML 2023 · similarity 0.86)
- Bayes-Optimal BER and AUC: Estimation and Evaluation of Estimators (NeurIPS 2026 · similarity 0.86)
- Bayes-optimal learning of an extensive-width neural network from quadratically many samples (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
