Research map / Learning theory and algorithms / Prediction and decision losses

Counterfactual Online Conformal Prediction Under Adaptive Logging

NeurIPS 2026 · Learning theory and algorithms › Prediction and decision losses · approach: Conformal prediction sets · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes propensity-weighted (and doubly robust) online conformal prediction that tracks counterfactual per-action coverage under endogenous logging, with theory and experiments on synthetic, Open Bandit, and DJIA data.

Problem

Online conformal prediction calibrated on logged data under adaptive logging keeps marginal coverage but miscovers counterfactual outcomes of rarely selected actions.

Approach

Replace the logged error indicator in the ACI recursion with an inverse-propensity-weighted (and doubly robust) estimator of counterfactual miscoverage per action, so the recursion's fixed point targets counterfactual rather than logged quantiles.

Key results

• On the mean-shift benchmark (K=2, Delta=2), PW-OCP reduces counterfactual gap to 0.017±0.007 and DR-OCP to 0.018±0.006, versus baseline gaps of 0.25-0.37.
• At K=20 arms, ACI has counterfactual gap 0.418±0.009 versus 0.074±0.007 for PW-OCP and 0.065±0.012 for DR-OCP.
• Online COPP gets the largest regret reduction (+94.9%), with PW-OCP at +91.1% and DR-OCP at +93.5% relative to ACI's 0%.

Surprising findings

• Online COPP achieves the largest decision-regret reduction in the table, slightly above PW-OCP and DR-OCP despite worse counterfactual coverage on several benchmarks.
• On the propensity-misspecification column, PW-OCP collapses to the ACI gap (0.252) while DR-OCP retains 0.030.

Limitations

• Positivity (pi_min > 0) is required; the framework excludes vanishing exploration.
• Sharpness and regret results require stationarity and local score-density regularity; DR guarantee is conditional on realized nuisance errors.
• Framework is contextual bandit rather than full RL with trajectory-level counterfactuals.

Experimental setup

Datasets: ZOZOTOWN Open Bandit Dataset (Women) ZOZOTOWN Open Bandit Dataset (Men) DJIA stock data

Benchmarks: Mean shift Variance shift Multi-arm Propensity error OBD Women OBD Men DJIA

Metrics: counterfactual coverage gap regret reduction vs ACI threshold error

Baselines: ACI FACI AgACI Conformal PID SAOCP NEx-CP ECI Online COPP

Builds on: adaptive conformal inference conformalized quantile regression off-policy conformal prediction doubly robust estimation

Closest papers by meaning

• Efficient Online Set-valued Classification with Bandit Feedback (ICML 2024 · similarity 0.87)
• Stochastic Online Conformal Prediction with Semi-Bandit Feedback (ICML 2025 · similarity 0.87)
• Online Conformal Prediction via Online Optimization (ICML 2025 · similarity 0.86)
• Adaptive Conformal Inference by Betting (ICML 2024 · similarity 0.86)
• Valid Selection among Conformal Sets (NeurIPS 2025 · similarity 0.86)
• Designing Decision Support Systems using Counterfactual Prediction Sets (ICML 2024 · similarity 0.85)