Research map / Learning theory and algorithms / Prediction and decision losses
Learning with Selectively Labeled Data from Multiple Decision-makers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies learning classifiers from selectively labeled data using multiple decision-makers as an instrumental variable, deriving exact and partial identification of classification risk and proposing a unified cost-sensitive learning (UCL) method.
Problem
Classification with selectively labeled data, where labels are observed only for units that received a positive historical decision, and the decision depends on unobserved confounders, causing selection bias in the risk.
Approach
Uses the assignment of heterogeneous decision-makers as an instrumental variable to identify classification risk, giving exact identification under a NUCEM assumption and sharp IV partial identification bounds otherwise, and reduces both to a unified weighted cost-sensitive learning problem.
Key results
- Partial Learning achieves gains over the Selected Sample baseline that grow as selection bias alpha increases (alpha in {0.5, 0.7, 0.9}), evaluated over 50 replications on the semi-synthetic loan data.
- Partial Learning outperforms Point Learning even under the NUCEM model where point identification holds, attributed to the difficulty of estimating the conditional variance ratio.
- On the synthetic multiclass dataset (K=3), the proposed methods, particularly Partial Learning, achieve higher accuracy than benchmarks under varying selection bias.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Partial Learning instead of Selected Sample | Semi-synthetic FICO dataset · FICO testing split | accuracy | Helped |
| Use Partial Learning instead of Point Learning | Semi-synthetic FICO dataset under NUCEM · FICO testing split | accuracy | Helped |
| Use Selected Sample-IPW instead of Selected Sample | Semi-synthetic FICO dataset · FICO testing split | accuracy | No clear effect |
| Increase the selection-bias parameter α | Point Learning and Partial Learning versus Selected Sample · FICO testing split | accuracy | Helped |
| Use Partial Learning instead of benchmark methods | Synthetic multiclass dataset · Synthetic dataset | accuracy | Helped |
Surprising findings
- Partial Learning performs better than Point Learning even when the point identification assumption holds, because estimating the conditional covariance ratio is hard in practice.
Limitations
- Exact identification requires the NUCEM assumption, which may be implausible in practice.
- Partial identification bounds are wide under the default assumption a=0, b=1, so the worst-case classifier can be conservative.
- Experiments are limited to a semi-synthetic loan dataset and a synthetic multiclass dataset, with the decision mechanism simulated.
Experimental setup
Datasets: FICO home loan dataset (semi-synthetic, 10459 observations) synthetic multiclass dataset (K=3)
Benchmarks: FICO home loan semi-synthetic dataset synthetic multiclass dataset
Metrics: classification accuracy on fully labeled test data
Baselines: Selected Sample Selected Sample-IPW Full Sample
Builds on: Balke and Pearl bounds Manski bounds NUCEM assumption (Kleinberg et al.) generalized random forests
Closest papers by meaning
- Towards a statistical theory of data selection under weak supervision (ICLR 2024 · similarity 0.85)
- Nearly Optimal Sample Complexity for Learning with Label Proportions (ICML 2025 · similarity 0.84)
- Accuracy vs. Accuracy: Computational Tradeoffs Between Classification Rates and Utility (NeurIPS 2026 · similarity 0.84)
- What Does It Take to Build a Performant Selective Classifier? (NeurIPS 2025 · similarity 0.84)
- One-Shot Strategic Classification Under Unknown Costs (ICML 2024 · similarity 0.84)
- Learning from positive and unlabeled examples -Finite size sample bounds (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.