AI Paper Atlas

Research map / Learning theory and algorithms / Prediction and decision losses

Should We Really Use Post-Hoc Tests Based on Mean-Ranks?

Alessio Benavoli, Giorgio Corani, Francesca Mangili

JMLR 2016 · Learning theory and algorithms › Prediction and decision losses · approach: Adaptive testing and experiments · 320 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add three strong algorithms C, D, E to the pool when comparing A and B with mean-ranks testMean-ranks post-hoc test · Example 1 (artificial, 20 datasets)significance of A vs B: not rejected with A,B alone; rejected with A,B,C,D,EMixed
add three much stronger algorithms C, D, E to the poolMean-ranks post-hoc test · Example 2 (A~N(0,1), B~N(1.5,1))power: 0.94 -> 0.046Hurt
include different pools of 2 extra classifiers when comparing C2 and C4Mean-ranks post-hoc test · Example 3 (UCI, 54 datasets)significance decisions: significant in 7 cases and not significant in 3 casesMixed
include different pools of 3 extra classifiers when comparing C2 and C4Mean-ranks post-hoc test · Example 3 (UCI, 54 datasets)significance decisions: significant in 9/10 casesMixed
switch the compared pool from C1-C4 to C1,C2,C4,C5Mean-ranks post-hoc test, C2 vs C4 · Example 3 (UCI, 54 datasets)z statistic vs threshold 2.64: 3.06 -> 2.46Mixed
use sign test or Wilcoxon signed-rank instead of mean-ranks post-hoc testpairwise comparison of classifiers · Example 2 and conclusionsindependence of decision from pool; power: power 0.046 -> 0.94Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy