AI Paper Atlas

Research map / Learning theory and algorithms / Prediction and decision losses

Good Classification Measures and How to Find Them

Martijn Gösgens, Anton Zhiyanov, Alexey Tikhonov, Liudmila Prokhorenkova

NeurIPS 2021 · Learning theory and algorithms › Prediction and decision losses · approach: Data attribution and recourse · 32 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use classification measures with more desirable properties, such as CC, SBA, or GM1, rather than measures with fewer propertiesBinary classification measure selection · Small binary datasets, n=2 through n=10Pairwise measure distinguishabilityImproved
Compare predictions using Accuracy rather than Balanced AccuracyRain-prediction threshold selection · Yandex.Weather rain predictionPairwise inconsistency rate (%): 96.5Mixed
Use F1 and GM1 rather than selecting thresholds by separate measuresRain-prediction threshold selection · Yandex.Weather rain predictionPairwise inconsistency rate (%): 2.2Improved
Use Cohen's Kappa rather than Balanced Accuracy for comparing classifiersRain-prediction threshold selection · Yandex.Weather rain predictionPairwise inconsistency rate (%): 58.9Mixed
Use Jaccard, Confusion Entropy, or SBA rather than Accuracy to rank modelsImageNet model comparison · ImageNet test setModel rankingMixed
Use macro-averaged CC rather than the standard multiclass CCMulticlass measure extension · ImageNet test setModel rankingMixed
Use Accuracy rather than Balanced Accuracy on balanced dataImageNet model comparison · ImageNet test setMeasure agreementNo clear effect
Use Correlation Distance, the arccosine transform of CC, when a distance interpretation is requiredClassification measure selection · Binary and multiclass classificationDistance propertyImproved
Use Generalized Means measures instead of requiring a metric distanceClassification measure selection · Binary classificationDesirable properties satisfiedImproved
Use Symmetric Balanced Accuracy for multiclass classificationClassification measure selection · Multiclass classificationDesirable properties preservedImproved
Require distance, monotonicity, and exact constant baseline simultaneouslyBinary classification measures · Binary classificationExistence of a measure satisfying all three propertiesHurt
Use macro averaging rather than micro or weighted averaging when averaging is necessaryMulticlass extensions of binary measures · Multiclass classificationProperties preservedImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy