Research map / Label-efficient and robust learning / Recognition and long-tail
Hidden Tails: Certifying Tail-Risk Claims under Selective Labels
El Mustapha Mansouri
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies certifying conditional value-at-risk (CVaR) tail-risk claims for predictors evaluated only on selectively observed labels, showing non-identifiability even under overlap.
Problem
High-stakes prediction systems are evaluated only on labels revealed by past decisions, so selective observation can hide the losses that determine full-population tail risk, making safe-looking predictors possibly unsafe in their worst compatible failures.
Approach
Proves that CVaR is non-identifiable under selective labels even with overlap, so tail-risk claims must be certified via bounds over the set of outcome distributions compatible with the observed data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase sample size from 10^3 to 10^5 (ablation) | oracle outer certificate · Synthetic finite-cell, Gamma=2, alpha=.75 | mean outer width: .614 -> .147 | Improved |
| increase sample size from 10^3 to 10^5 (ablation) | exact population (identification) width · Synthetic finite-cell, Gamma=2, alpha=.75 | mean exact population width: .0949 constant | No clear effect |
| increase sample size from 10^3 to 10^5 (ablation) | cross-fitted endpoint estimator · Synthetic finite-cell, Gamma=2, alpha=.75 | upper-endpoint RMSE: .0361 -> .0027 | Improved |
| assume Gamma=1 (conditional ignorability) instead of Gamma=2 sensitivity | default-weighted logistic regression vs others · LendingClub 2012, alpha=.95 | pairwise model ordering: two of three pairwise orderings reverse; .661 accepted-only -> .813 applicant | Hurt |
| add fold-variability enlargement to plug-in intervals (ablation) | plug-in CVaR intervals · Masked issued loans, Gamma0=3 | coverage of reference CVaR (of 20 masks): 11-14 -> 19-20 | Improved |
| use Gamma=1 ordering rule instead of Gamma=3 interval separation (ablation) | predictor comparison · Masked issued loans, both tail levels | ordering errors: 16 errors out of 120 declared orderings (vs 37 correct, 23 unresolved at Gamma=3) | Hurt |
| use TIV per-cost targeted acquisition instead of random (ablation) | fixed-threshold audit, boosting · Masked issued loans, alpha=.95, Gamma=3 | median fraction of initial gap removed: 16% -> 61% | Improved |
| use TIV acquisition instead of random in sequential retraining (ablation) | boosting · Masked issued loans, alpha=.95, Gamma=3 | terminal median held-out CVaR: .8682 -> .8541 | Improved |
| use TIV acquisition instead of random in sequential retraining (ablation) | boosting · Masked issued loans, alpha=.95, Gamma=3 | mean Brier loss: .1347 -> .1362 | Hurt |
| sequential TIV acquisition with retraining | boosting · Masked issued loans, alpha=.95, Gamma=3 | median estimated retained width: .097 -> .035 | Improved |
Surprising findings
- CVaR is non-identifiable under selective labels even when overlap holds.
Closest papers by meaning
- Adversarially Robust Control of Conditional Value-at-Risk via Rockafellar-Uryasev Conformal Inference (ICML 2026 · similarity 0.86)
- Active, anytime-valid risk controlling prediction sets (NeurIPS 2024 · similarity 0.83)
- Decision Theoretic Foundations for Conformal Prediction: Optimal Uncertainty Quantification for Risk-Averse Agents (ICML 2025 · similarity 0.82)
- Conformal Risk Training: End-to-End Optimization of Conformal Risk Control (NeurIPS 2025 · similarity 0.82)
- Rethinking Confidence Scores and Thresholds in Pseudolabeling-based SSL (ICML 2025 · similarity 0.82)
- Rethinking Guidance Information to Utilize Unlabeled Samples: A Label Encoding Perspective (ICML 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
