Research map / Learning theory and algorithms / Optimal transport
Multivariate Stochastic Dominance via Optimal Transport and Applications to Models Benchmarking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces an entropically regularized, OT-based multivariate almost-first-order-stochastic-dominance violation ratio with a CLT and bootstrap consistency, and applies it to ranking LLMs on multiple metrics.
Problem
Multivariate stochastic dominance testing is underdeveloped; existing LLM benchmarking reduces multi-metric comparisons to univariate orders, ignoring metric dependencies, and exact empirical OT suffers from the curse of dimensionality and high computational cost.
Approach
Characterize multivariate first-order stochastic dominance via couplings, define a violation ratio as a ratio of OT costs with smooth costs (e.g. logistic surrogate of the 0/1 loss), regularize with entropic OT, and prove a CLT and bootstrap consistency via the functional delta method to enable hypothesis tests solved by Sinkhorn.
Key results
- On MixInstruct with d=9 metrics, the entropic multivariate FSD ranking achieves the highest Kendall tau with the ChatGPT-based ranking, outperforming the independent-copula and empirical-copula aggregation baselines in sample efficiency.
- In the synthetic experiment, the multivariate violation ratio varies approximately linearly with the dominance parameter p, and the entropic logistic-cost ratio converges to the unregularized hinge baseline as lambda decreases and beta increases.
- The paper proves a CLT for the entropic violation ratio and bootstrap consistency under sub-Gaussian marginals with smooth costs satisfying (SC_d).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use entropic multivariate FSD instead of univariate FSD aggregation | LLM rankings on MixInstruct test · MixInstruct test | Kendall tau rank similarity with GPT score | Helped |
| Use independent-copula portfolio aggregation instead of entropic multivariate FSD | LLM rankings on MixInstruct test · MixInstruct test | sample efficiency | Hurt |
| Use empirical-copula portfolio aggregation instead of entropic multivariate FSD | LLM rankings on MixInstruct test · MixInstruct test | sample efficiency | Hurt |
| Use logistic-cost entropic multivariate FSD with decreasing regularization and increasing gain | Synthetic dominance estimation · Synthetic data | approximation to unregularized hinge-cost violation ratio | Helped |
Limitations
- Hinge (squared) cost is not smooth enough for the CLT in arbitrary dimension, so the smooth surrogate (logistic) must be used.
- Bonferroni FWER correction is conservative and gives uniform but low power across tests.
- LLM ranking uses ChatGPT scores as the reference ground truth rather than human judgments.
Experimental setup
Datasets: MixInstruct
Benchmarks: MixInstruct
Metrics: Kendall tau rank similarity with ChatGPT score violation ratio bootstrap confidence intervals
Baselines: univariate FSD on ChatGPT scores portfolio aggregation with independent copula portfolio aggregation with empirical copula unregularized OT (EMD) violation ratio
Builds on: almost stochastic dominance (Dolati/Mehrotra-style OT violation ratio) entropic optimal transport functional delta method Sinkhorn algorithm
Closest papers by meaning
- Distributional Preference Alignment of LLMs via Optimal Transport (NeurIPS 2024 · similarity 0.85)
- Multidimensional Bayesian Utility Maximization: Tight Approximations to Welfare (NeurIPS 2025 · similarity 0.84)
- Risk Aware Benchmarking of Large Language Models (ICML 2024 · similarity 0.84)
- The Bias-Variance Tradeoff in Data-Driven Optimization: A Local Misspecification Perspective (NeurIPS 2025 · similarity 0.83)
- Model Alignment as Prospect Theoretic Optimization (ICML 2024 · similarity 0.83)
- Aggregating Quantitative Relative Judgments: From Social Choice to Ranking Prediction (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.