Research map / Probabilistic and scientific modeling / Treatment effects
Rethinking Causal Ranking: A Balanced Perspective on Uplift Model Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that uplift and Qini curves are biased by ignoring negative outcomes, proposes the Principled Uplift Curve (PUC) evaluation metric and the Principled Uplift Loss used in a DragonNet-based PTONet model.
Problem
Conventional uplift and Qini curves only count individuals with positive outcomes, so they can reward biased CATE rankings over unbiased ones and lead to suboptimal model selection when counterfactuals are unobservable.
Approach
Define a balanced curve (PUC) that gives equal value to persuadables with positive and negative outcomes (TP and CN) and to sleeping dogs (TN and CP), and train uplift models with a BCE-based Principled Uplift Loss derived from it, combined with a DragonNet-style three-head network (PTONet).
Key results
- On synthetic data, PUC achieves Kendall correlation of 1 with AUTGC while SUC, SQC, JUC and JQC do not fully align.
- In the case study, the biased model tau(2) gets AUUQC 1.000 under SUC and JQC versus 0.500 for the unbiased tau(1), while PUC reverses this (0.613 vs 0.484).
- On Criteo, PTONet attains PUC AUUQC 0.191±0.01, the highest among the compared models (EUEN 0.186, DragonNet 0.137).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use PTONet instead of the strongest listed baseline, EUEN | Uplift modeling · Criteo (visits) | PUC AUUQC: 0.186±0.01 -> 0.191±0.01 | Helped |
| Add PUL loss to PTONet (ablation) | PTONet · Synthetic data | PUC AUUQC: 0.659±0.45 -> 0.948±0.15 | Helped |
| Add PUL loss to PTONet (ablation) | PTONet · Synthetic data | AUTGC: 0.739±0.35 -> 0.961±0.11 | Helped |
| Add balance loss LB to PTONet (ablation) | PTONet · Synthetic data | PUC AUUQC: 0.678±0.46 -> 0.948±0.15 | Helped |
| Add treatment loss LT to PTONet (ablation) | PTONet · Synthetic data | PUC AUUQC: 0.756±0.41 -> 0.948±0.15 | Helped |
| Remove targeted regularizer LT R from PTONet (ablation) | PTONet · Synthetic data | PUC AUUQC: 0.948±0.15 -> 0.936±0.22 | Hurt |
| Use PUL-guided S-Learner instead of standard S-Learner | S-Learner · Synthetic data | PUC AUUQC: 0.609±0.42 -> 0.943±0.19 | Helped |
| Evaluate rankings with PUC rather than conventional uplift and Qini curves | Uplift model evaluation · Synthetic data | Kendall correlation with AUTGC: Kendall = 1 | Helped |
Surprising findings
- Conventional uplift and Qini curves can rank a biased model above an unbiased one, and can even produce the maximum AUUC value under incorrect rankings.
- Conventional curves can still give biased evaluations even on RCT data, contrary to prior work suggesting they are reliable there.
Limitations
- PUC does not fully distinguish among all four principal strata (sure things and lost causes are not separated).
- Validation of PUC's alignment with true CATE ranking relies on synthetic data where true effects are known; real-world evaluation lacks ground truth.
- PTONet's gains on real data are judged only by the PUC metric itself, which the method is designed to optimize.
Experimental setup
Datasets: synthetic data (n=10,000, q=10) Criteo uplift dataset Lazada dataset
Benchmarks: synthetic data with known CATE Criteo Lazada
Metrics: AUUQC PUC SUC SQC JUC JQC PEHE AUTGC Kendall correlation
Baselines: S-Learner T-Learner TARNet CFRNet DragonNet EUEN DESCN EFIN
Builds on: DragonNet Qini curve uplift curve Devriendt et al. 2020 uplift helper loss
Closest papers by meaning
- Invariant Deep Uplift Modeling for Incentive Assignment in Online Marketing via Probability of Necessity and Sufficiency (ICML 2025 · similarity 0.88)
- RATE: Causal Explainability of Reward Models with Imperfect Counterfactuals (ICML 2025 · similarity 0.83)
- Bi-Level Decision-Focused Causal Learning for Large-Scale Marketing Optimization: Bridging Observational and Experimental Data (NeurIPS 2025 · similarity 0.83)
- Doubly robust identification of treatment effects from multiple environments (ICLR 2025 · similarity 0.82)
- Empirical Analysis of Model Selection for Heterogeneous Causal Effect Estimation (ICLR 2024 · similarity 0.82)
- Learning Counterfactual Outcomes Under Rank Preservation (NeurIPS 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.