Research map / Learning theory and algorithms / Prediction and decision losses

A Survey of Learning Criteria Going beyond the Usual Risk (Abstract Reprint)

AAAI 2024 · Learning theory and algorithms › Prediction and decision losses · approach: Losses for decision-making · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Survey and unifying framework that generalizes the classical expected-loss learning paradigm to arbitrary learning criteria (properties of the loss distribution), covering robust, order-based, OCE, and DRO risks.

Problem

Machine learning typically optimizes the expected (average) loss, which ignores tails, fairness, robustness, and distribution shift; the paper asks what makes a desirable loss distribution beyond its mean.

Approach

Generalizes the learning problem by introducing a learning criterion mapping the random test-loss distribution to a real value and a desirability set, then surveys and organizes criteria (order-based/L-risks, OCE/tilted/CVaR, DRO/f-divergence risks, robust M-estimation) as special cases within this single formulation.

Key results

• Shows the standard expected-loss setup is a special case of the generalized learning criterion formulation with desirability set C = (-inf, threshold].
• Shows OCE risks always upper-bound the expected loss, and inverted OCE is always at most the mean.
• Shows the mean-variance criterion is not a valid OCE risk, correcting an error in prior literature.

Surprising findings

• Mean-variance criterion is not a valid OCE risk despite being listed as one in prior work (Table 1 of a cited paper).
• CVaR on Bernoulli training losses collapses to expected 0-1 error.

Limitations

• Explicitly omits reinforcement learning, Bayesian learning theory, duality theory, and adversarial methods.
• Focuses purely on the criterion; loss design, regularization and hypothesis constraints are abstracted away.
• Computational tractability of many criteria (e.g., robust M-estimators, mean-variance) is challenging and largely left abstract.

Experimental setup

Builds on: empirical risk minimization Haussler's learning model optimized certainty equivalent risk quadrangle

Closest papers by meaning

• Leaving the Nest: Going beyond Local Loss Functions for Predict-Then-Optimize (AAAI 2024 · similarity 0.89)
• Random Classification Noise does not defeat All Convex Potential Boosters Irrespective of Model Choice (ICML 2023 · similarity 0.86)
• Toward a Characterization of Loss Functions for Distribution Learning (NeurIPS 2019 · similarity 0.86)
• On the Efficiency of ERM in Feature Learning (NeurIPS 2024 · similarity 0.86)
• Generalization Analysis of Machine Learning Algorithms via the Worst-Case Data-Generating Probability Measure (AAAI 2024 · similarity 0.85)
• Robust Loss Functions for Training Decision Trees with Noisy Labels (AAAI 2024 · similarity 0.84)