Research map / Learning theory and algorithms / Prediction and decision losses
The Value of Prediction in Identifying the Worst-Off
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Develops a framework and prediction-access ratio (PAR) to compare the welfare value of improving prediction versus expanding screening capacity for identifying the worst-off, with theory under Gaussian assumptions and a German long-term unemployment case study.
Problem
How to assess the relative value of improving risk prediction versus expanding screening capacity when a public agency aims to identify the worst-off members of a population.
Approach
Adapts the prediction-access ratio (PAR) from allocation problems to worst-off identification, measuring fraction of truly at-risk individuals screened; derives Gaussian closed forms and bounds showing prediction improvements matter most at very low and very high R^2, while expanding access dominates when capacity is scarce.
Key results
- CatBoost achieves R^2 of 0.15 on the 2015 test set; policy value of about 0.28 at alpha about 0.1 and 0.56 at alpha about 0.25 for beta about 0.15.
- Screening capacity must rise by about 0.25 above beta to screen 75% of high-risk jobseekers, consistently across beta values.
- Theory: PAR goes to zero in the local regime as R^2 approaches 0 or 1 (with alpha=beta at high R^2), and PAR is at least 1 for alpha <= beta under moderate R^2 bounds.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase screening capacity from α ≈ 0.1 to α ≈ 0.25 | CatBoost · German LTU | policy value: 0.28 and 0.56 | Helped |
| increase screening capacity | CatBoost · German jobseekers, LTU cutoff (β ≈ 0.15) | policy value | Helped |
| increase screening capacity by Δα = 0.1 rather than improve prediction by ΔR² = 0.1 | CatBoost with R² = 0.15 · German jobseekers | prediction-access ratio: PAR > 1 | Helped |
| increase screening capacity by Δα = 0.1 rather than improve prediction by ΔR² = 0.1 | constant predictions (R² = 0) · German jobseekers | prediction-access ratio: PAR < 1 for α ≥ 0.1 | Mixed |
| increase screening capacity by Δα = 0.1 rather than improve prediction by ΔR² = 0.1 | near-perfect predictions (R² = 0.9) · German jobseekers | prediction-access ratio: R² = 0.9 | Mixed |
| replace CatBoost with a 4-depth decision tree (ablation) | CatBoost · German jobseekers | R²: 5% decrease in R² | Hurt |
| replace CatBoost with a 4-depth decision tree (ablation) | CatBoost · German jobseekers | policy value: 1–8% reduction in policy value | Hurt |
| reframe unemployment-duration regression as binary classification (ablation) | CatBoost · German jobseekers test set | policy value | No clear effect |
| scale CatBoost residuals to increase R² by 0.1 (ablation) | CatBoost · German jobseekers | prediction-access ratio: ΔR² = Δα = 0.1 | Mixed |
| increase the amount of training data (ablation) | CatBoost · German jobseekers test set | R² | Helped |
Surprising findings
- Under random allocation (R^2=0) with Delta=0.1, the PAR stays below one for alpha >= 0.1, so prediction improvements can matter more than capacity at larger screening levels.
- A simple 4-depth decision tree loses only 1-8% policy value relative to CatBoost despite a 5% R^2 drop.
Limitations
- Theoretical results rely on Gaussian errors independent of predictions; empirical R^2 improvements simulated by uniform residual scaling.
- Costs are not modeled in the empirical analysis; PAR is presented separately from costs.
- Single case study on German unemployment data with a single model family for prediction.
Experimental setup
Datasets: Sample of Integrated Employment Biographies (SIAB) 2% sample of German administrative labor market records
Benchmarks: SIAB long-term unemployment test set (2015)
Metrics: policy value (recall of worst-off) prediction-access ratio R^2 policy value difference
Baselines: shallow decision tree (max_depth 4) random allocation (R^2=0)
Builds on: prediction-access ratio (Perdomo et al.) prediction policy problems
Closest papers by meaning
- The Relative Value of Prediction in Algorithmic Decision Making (ICML 2024 · similarity 0.86)
- Allocation Requires Prediction Only if Inequality Is Low (ICML 2024 · similarity 0.85)
- Predictive Performance Comparison of Decision Policies Under Confounding (ICML 2024 · similarity 0.84)
- The Hidden Cost of Waiting for Accurate Predictions (ICLR 2025 · similarity 0.84)
- Policy Learning for Balancing Short-Term and Long-Term Rewards (ICML 2024 · similarity 0.83)
- Mind the Gap: A Causal Perspective on Bias Amplification in Prediction & Decision-Making (NeurIPS 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.