Research map / Learning theory and algorithms / Prediction and decision losses
Statistical Multicriteria Benchmarking via the GSD-Front
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes the GSD-front, a generalized stochastic dominance based multicriteria classifier comparison set with consistent estimator, permutation tests (static/dynamic), and contamination-robustness checks, illustrated on OpenML and PMLB.
Problem
Comparing classifiers under multiple quality metrics (cardinal and ordinal) over a benchmark suite is hard: weighted sums need arbitrary weights and Pareto fronts are uninformative, while statistical uncertainty from the choice of data sets and i.i.d. violations are often ignored.
Approach
Compare classifiers via generalized stochastic dominance over a preference system that encodes both ordinal and cardinal metric information, define the GSD-front as the set of non-dominated classifiers, estimate it via an epsilon-empirical GSD-front with a permutation-test-based statistical test, and robustify the test against contaminated samples using gamma-contamination models from imprecise pr
Key results
- On OpenML, the 0-empirical GSD-front is formed by RF, CART and kNN; CART strictly empirically GSD-dominates xGBoost, SVM, LR and GLMNet.
- SVM significantly lies in the GSD-front of the subset {SVM, kNN, xGBoost, RF, GLMNet} at alpha=0.05; the decision remains stable under contamination of up to 7 of 80 datasets.
- First-order stochastic dominance yields no significant differences, and the Pareto front contains all classifiers on OpenML, so the GSD-front is more informative.
- On PMLB, no evidence is found to reject that the new CRE classifier is dominated by the five state-of-the-art classifiers under robust accuracy.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the GSD-front instead of the Pareto-front for multicriteria comparison | OpenML classifier comparison · OpenML 80 binary classification datasets | front informativeness: all considered classifiers -> RF, CART, and kNN | Helped |
| Use GSD with cardinal accuracy rather than first-order stochastic dominance | OpenML classifier comparison · OpenML 80 binary classification datasets | significant classifier comparisons | Helped |
| Test SVM against competitors using the dynamic GSD-test | SVM · OpenML 80 binary classification datasets | GSD-front membership significance: α~=~0.05 | Helped |
| Allow contamination of the benchmark suite | SVM dynamic GSD-test · OpenML 80 binary classification datasets | GSD-front membership significance: 7 out of 80 data sets | Hurt |
| Compare CRE against the state-of-the-art classifiers using the GSD-test | CRE · PMLB 62 datasets | GSD-front membership significance: α = 0.05 | No clear effect |
Surprising findings
- The Pareto front on OpenML contains all considered classifiers, showing Pareto analysis is uninformative in this setting.
- The marginal (single-metric) front on OpenML consists of all classifiers, also less informative than the GSD-front.
Limitations
- Static GSD-test is conservative because it ignores correlations among pairwise tests and robustness analysis is worst-case.
- Does not account for meta-properties of data sets (covariates); only classifiers compared, not deep learning models.
- Assumes i.i.d. sampling of data sets for validity of the base tests, with robustness only checked via contamination.
Experimental setup
Datasets: OpenML binary classification suite (80 datasets) PMLB (62 datasets)
Benchmarks: OpenML binary classification suite PMLB
Metrics: predictive accuracy train computation time (ordinal) test computation time (ordinal) robust accuracy (noisy features/classes)
Baselines: first-order stochastic dominance Pareto front marginal front (single-metric tests)
Builds on: generalized stochastic dominance Pareto front permutation tests for stochastic dominance
Closest papers by meaning
- Class-Attribute Priors: Adapting Optimization to Heterogeneity and Fairness Objective (AAAI 2024 · similarity 0.83)
- Leveraging Uncertainty Estimates To Improve Classifier Performance (ICLR 2024 · similarity 0.81)
- How many classifiers do we need? (NeurIPS 2024 · similarity 0.81)
- Stability and Multigroup Fairness in Ranking with Uncertain Predictions (ICML 2024 · similarity 0.81)
- Exact power indices for plurality-voting ensembles (NeurIPS 2026 · similarity 0.81)
- Principled Algorithms for Optimizing Generalized Metrics in Binary Classification (ICML 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.