Research map / Learning theory and algorithms / Prediction and decision losses

Improving Model Probability Calibration by Integration of Large Data Sources with Biased Labels

AAAI 2025 · Learning theory and algorithms › Prediction and decision losses · approach: Distributionally robust optimization · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CalEM, an approximate-EM procedure that lets logistic-likelihood probability calibrators use a large cheap but treatment-biased label source alongside a small clean calibration set.

Problem

Probability calibration for rare events is unstable when only a small expensive clean labeled set is available, while a large cheap data source has biased labels (treatment-induced in marketing).

Approach

Treat the unobserved counterfactual clean labels in the biased treatment group as latent variables, use a surrogate conditional distribution learned from the biased labels in an approximate E-step, and iteratively refit any logistic-likelihood calibrator with EM-derived weights; convergence is inherited from approximate EM.

Key results

• In simulation, the proposed method reduced L2 calibration error relative to GAM by up to about 52% (e.g., +52.82% in one setting), with improvements most pronounced for mild label imbalance and small clean sets.
• On real marketing data, BC-EM achieved the best KS error on Criteo (0.000190 vs 0.000224 for BC) and Lenta (0.001918 vs 0.002073 for BC), and the best Brier/log-likelihood in most columns.
• CalEM consistently improved its base calibrators (Beta calibration and GAM) across the three marketing datasets over 10 re-runs.

Measured effects reported in the paper

Surprising findings

• Simulation gains become negligible or negative in some settings (e.g., -3.62% at 30% observed data with g1, p=0.5 under h2 smoothness), and when the clean sample is large (50%) improvements vanish.

Limitations

• Requires an assumption on treatment effect form (h(X) from P(Y=0|X)), which relies on the monotone/non-stringent condition P(Y=0|f(X)=0)<1.
• Convergence guarantee is local and relies on the surrogate distribution being closer to the exact one than to the true one.
• Evaluated only on marketing treatment-control data and simulations.

Experimental setup

Datasets: Criteo Hillstrom Lenta simulated data

Benchmarks: Criteo Hillstrom Lenta simulation L2 calibration error

Metrics: Kolmogorov-Smirnov error Brier score log-likelihood L2 error

Baselines: GAM Beta calibration Isotonic regression BC-EM ablation GAM-EM ablation

Builds on: Expectation-Maximization (Dempster, Laird, Rubin 1977) GAM spline calibration (Gupta et al. 2021) Beta calibration (Kull et al. 2017) Isotonic regression

Closest papers by meaning

• Testing Calibration in Nearly-Linear Time (NeurIPS 2024 · similarity 0.84)
• Calibration without labels in multiple testing (NeurIPS 2026 · similarity 0.83)
• Combining Priors with Experience: Confidence Calibration Based on Binomial Process Modeling (AAAI 2025 · similarity 0.83)
• Reassessing How to Compare and Improve the Calibration of Machine Learning Models (ICLR 2025 · similarity 0.83)
• Algorithms with Calibrated Machine Learning Predictions (ICML 2025 · similarity 0.83)
• The Minimax Rate of Second-Order Calibration (NeurIPS 2026 · similarity 0.83)