Research map / Probabilistic and scientific modeling / Uncertainty and Gaussian processes
Time for a Change: a Tutorial for Comparing Multiple Classifiers Through Bayesian Analysis
Alessio Benavoli, Giorgio Corani, Janez Demšar, Marco Zaffalon
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace NHST with Bayesian correlated t-test with rope | classifier comparison nbc vs aode · 54 UCI data sets | data sets with a decision: 19 -> 21 | Improved |
| add a rope to the Bayesian analysis of NHST non-rejections | nbc vs aode · 54 UCI data sets | practical equivalence declared: 6 of 35 | Improved |
| use Bayesian correlated t-test with rope on all pairs | nbc, aode, hnb, j48, j48gr · 540 comparisons (NHST non-rejections) | practically equivalent decisions: 74/341= 0.22 | Improved |
| use Bayesian test instead of NHST for significant comparisons | five classifiers · 540 comparisons (NHST rejections) | confirmed differences: 142/199 confirmed, 51/199 no decision, 6/199 equivalent | Mixed |
| vary Dirichlet prior location z0 to -infinity or +infinity (ablation) | Bayesian signed-rank test · nbc vs aode, hnb, j48, j48gr | posterior probabilities: 0.897 vs 0.888 vs 0.904 (nbc-aode right) | No clear effect |
| use hierarchical Bayesian model over degrees-of-freedom prior (ablation) | Bayesian correlated t-test · UCI data sets | sensitivity to nu prior | Improved |
Closest papers by meaning
- Full Bayesian Significance Testing for Neural Networks (AAAI 2024 · similarity 0.84)
- Recursive PAC-Bayes: A Frequentist Approach to Sequential Prior Updates with No Information Loss (NeurIPS 2024 · similarity 0.83)
- Exploiting LLMs for Automatic Hypothesis Assessment via a Logit-Based Calibrated Prior (NeurIPS 2025 · similarity 0.83)
- Calibration without labels in multiple testing (NeurIPS 2026 · similarity 0.83)
- Reproduce, Replicate, Reevaluate. The Long but Safe Way to Extend Machine Learning Methods (AAAI 2024 · similarity 0.83)
- Task-Agnostic Machine-Learning-Assisted Inference (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
