Research map / Learning theory and algorithms / Prediction and decision losses
Should We Really Use Post-Hoc Tests Based on Mean-Ranks?
Alessio Benavoli, Giorgio Corani, Francesca Mangili
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add three strong algorithms C, D, E to the pool when comparing A and B with mean-ranks test | Mean-ranks post-hoc test · Example 1 (artificial, 20 datasets) | significance of A vs B: not rejected with A,B alone; rejected with A,B,C,D,E | Mixed |
| add three much stronger algorithms C, D, E to the pool | Mean-ranks post-hoc test · Example 2 (A~N(0,1), B~N(1.5,1)) | power: 0.94 -> 0.046 | Hurt |
| include different pools of 2 extra classifiers when comparing C2 and C4 | Mean-ranks post-hoc test · Example 3 (UCI, 54 datasets) | significance decisions: significant in 7 cases and not significant in 3 cases | Mixed |
| include different pools of 3 extra classifiers when comparing C2 and C4 | Mean-ranks post-hoc test · Example 3 (UCI, 54 datasets) | significance decisions: significant in 9/10 cases | Mixed |
| switch the compared pool from C1-C4 to C1,C2,C4,C5 | Mean-ranks post-hoc test, C2 vs C4 · Example 3 (UCI, 54 datasets) | z statistic vs threshold 2.64: 3.06 -> 2.46 | Mixed |
| use sign test or Wilcoxon signed-rank instead of mean-ranks post-hoc test | pairwise comparison of classifiers · Example 2 and conclusions | independence of decision from pool; power: power 0.046 -> 0.94 | Improved |
Closest papers by meaning
- Identity testing for Mallows model (NeurIPS 2021 · similarity 0.81)
- Permutation-based Rank Test in the Presence of Discretization and Application in Causal Discovery with Mixed Data (ICML 2025 · similarity 0.81)
- YEAST: Yet Another Sequential Test (NeurIPS 2025 · similarity 0.79)
- A hierarchical decomposition for explaining ML performance discrepancies (NeurIPS 2024 · similarity 0.79)
- Statistical Multicriteria Benchmarking via the GSD-Front (NeurIPS 2024 · similarity 0.79)
- Boosting Test Performance with Importance Sampling--a Subpopulation Perspective (AAAI 2025 · similarity 0.79)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
