Research map / Learning theory and algorithms / Prediction and decision losses
Good Classification Measures and How to Find Them
Martijn Gösgens, Anton Zhiyanov, Alexey Tikhonov, Liudmila Prokhorenkova
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use classification measures with more desirable properties, such as CC, SBA, or GM1, rather than measures with fewer properties | Binary classification measure selection · Small binary datasets, n=2 through n=10 | Pairwise measure distinguishability | Improved |
| Compare predictions using Accuracy rather than Balanced Accuracy | Rain-prediction threshold selection · Yandex.Weather rain prediction | Pairwise inconsistency rate (%): 96.5 | Mixed |
| Use F1 and GM1 rather than selecting thresholds by separate measures | Rain-prediction threshold selection · Yandex.Weather rain prediction | Pairwise inconsistency rate (%): 2.2 | Improved |
| Use Cohen's Kappa rather than Balanced Accuracy for comparing classifiers | Rain-prediction threshold selection · Yandex.Weather rain prediction | Pairwise inconsistency rate (%): 58.9 | Mixed |
| Use Jaccard, Confusion Entropy, or SBA rather than Accuracy to rank models | ImageNet model comparison · ImageNet test set | Model ranking | Mixed |
| Use macro-averaged CC rather than the standard multiclass CC | Multiclass measure extension · ImageNet test set | Model ranking | Mixed |
| Use Accuracy rather than Balanced Accuracy on balanced data | ImageNet model comparison · ImageNet test set | Measure agreement | No clear effect |
| Use Correlation Distance, the arccosine transform of CC, when a distance interpretation is required | Classification measure selection · Binary and multiclass classification | Distance property | Improved |
| Use Generalized Means measures instead of requiring a metric distance | Classification measure selection · Binary classification | Desirable properties satisfied | Improved |
| Use Symmetric Balanced Accuracy for multiclass classification | Classification measure selection · Multiclass classification | Desirable properties preserved | Improved |
| Require distance, monotonicity, and exact constant baseline simultaneously | Binary classification measures · Binary classification | Existence of a measure satisfying all three properties | Hurt |
| Use macro averaging rather than micro or weighted averaging when averaging is necessary | Multiclass extensions of binary measures · Multiclass classification | Properties preserved | Improved |
Closest papers by meaning
- Binary Classification with Karmic, Threshold-Quasi-Concave Metrics (ICML 2018 · similarity 0.85)
- Foundations of the Theory of Performance-Based Ranking (CVPR 2025 · similarity 0.82)
- Accuracy vs. Accuracy: Computational Tradeoffs Between Classification Rates and Utility (NeurIPS 2026 · similarity 0.82)
- Simple Weak Coresets for Non-decomposable Classification Measures (AAAI 2024 · similarity 0.81)
- Principled Algorithms for Optimizing Generalized Metrics in Binary Classification (ICML 2025 · similarity 0.81)
- How many classifiers do we need? (NeurIPS 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
