Research map / Learning theory and algorithms / Kernels and regression theory
Adjust Pearson's $r$ to Measure Arbitrary Monotone Dependence
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes rearrangement correlation, a tighter-bound adjustment of Pearson's r derived from a sharpened Cauchy-Schwarz inequality via rearrangement theorems, that measures arbitrary monotone dependence and reduces to r in linear cases.
Problem
Pearson's r is widely thought to capture only linear dependence, and existing nonlinear measures (Spearman, Kendall, HSIC, dCor, MIC, Chatterjee's xi) do not use the Pearson/covariance framework; the paper asks whether Pearson-style scaled covariance can accurately measure nonlinear monotone depende
Approach
Scaling covariance by a tighter bound (the rearranged covariance |cov(X↑, Y↕)| instead of sqrt(var X var Y)) from a new inequality sharper than Cauchy-Schwarz enlarges the capture range from linear to arbitrary monotone dependence, so r# = cov / |cov(X↑,Y↕)| reverts to Pearson's r in linear cases.
Key results
- In 50 simulated monotone scenarios, rearrangement correlation has the lowest MAE (0.060) versus Spearman's rho (0.102), dCor (0.127) and Pearson's r (0.150).
- On five NIST real-life monotone scenarios, rearrangement correlation achieves MAE 0.00141 versus Spearman's rho 0.0159 and Pearson's r 0.0916.
- On 16 non-monotone scenarios, rearrangement correlation (MAE 0.418) is worse than xi (0.141), MIC (0.157) and dCor (0.364), but beats Spearman (0.431), Pearson (0.437) and Kendall (0.461).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Pearson's r with Rearrangement Correlation | Dependence-strength measurement · 50 simulated monotone scenarios | MAE: 0.150 -> 0.060 | Helped |
| Replace Pearson's r with Rearrangement Correlation | Dependence-strength measurement · 5 NIST real-life monotone scenarios | MAE: 0.0916 -> 0.00141 | Helped |
| Use Rearrangement Correlation instead of Chatterjee's xi | Dependence-strength measurement · 16 non-monotone scenarios | MAE: 0.141 -> 0.418 | Hurt |
| Use Rearrangement Correlation instead of Pearson's r | Dependence-strength measurement · 16 non-monotone scenarios | MAE: 0.437 -> 0.418 | Helped |
Surprising findings
- Rearrangement correlation outperforms Spearman's rho in monotone scenarios despite Spearman's rank-based approach being the standard monotone measure.
- Tighter bounds on covariance expand (rather than shrink) the capture range, the opposite of the intuition that tighter bounds restrict scope.
Experimental setup
Datasets: NIST nonlinear regression datasets (Chwirut1, Hahn1, Rat43, Roszman1, Thurber)
Benchmarks: 50 simulated monotone scenarios NIST real-life monotone scenarios (5) 16 non-monotone scenarios
Metrics: mean absolute error (MAE) trueness precision
Baselines: Pearson's r Spearman's rho Kendall's tau HSIC distance correlation MIC Chatterjee's xi Additivity coefficient (r+)
Builds on: Pearson correlation Cauchy-Schwarz inequality rearrangement inequality Concordance correlation coefficient Additivity coefficient
Closest papers by meaning
- SHGR: A Generalized Maximal Correlation Coefficient (NeurIPS 2025 · similarity 0.83)
- Measuring Dependence with Matrix-based Entropy Functional (AAAI 2021 · similarity 0.83)
- TIC-TAC: A Framework For Improved Covariance Estimation In Deep Heteroscedastic Regression (ICML 2024 · similarity 0.81)
- Identification of Causal Structure with Latent Variables Based on Higher Order Cumulants (AAAI 2024 · similarity 0.81)
- A Skewness-Based Criterion for Addressing Heteroscedastic Noise in Causal Discovery (ICLR 2025 · similarity 0.81)
- S$\Omega$I: Score-based O-INFORMATION Estimation (ICML 2024 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.