Research map / Reinforcement learning / Policy gradient and actor-critic

Conformal Prediction Beyond the Horizon: Distribution-Free Inference for Policy Evaluation

NeurIPS 2025 · Reinforcement learning › Policy gradient and actor-critic · approach: Distributional value learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a conformal prediction framework for infinite-horizon policy evaluation that builds k-step truncated pseudo-returns from a QTD distributional model, calibrates via experience replay with (weighted) subsampling, and gives Wasserstein-based coverage lower bounds.

Problem

Constructing distribution-free prediction intervals for infinite-horizon discounted returns in on-policy and off-policy RL, where returns are unobserved beyond a finite horizon, data are temporally dependent, and distribution shifts violate exchangeability.

Approach

Wrap a distributional RL estimator (QTD) with split conformal prediction using k-step pseudo-returns (observed rewards plus a sampled bootstrapped tail), decorrelate calibration data via experience replay, and restore approximate exchangeability with importance-weighted subsampling; coverage gap is bounded via Wasserstein distance with a gamma^k factor.

Key results

• On synthetic examples at 90% nominal level, the proposed k-step conformal intervals (k=1..5) achieve near-nominal coverage across on-policy and off-policy settings over 50 repetitions, while DRL-QR undercovers due to model bias.
• Average interval length increases with k, reflecting higher variance from longer truncation horizons.
• In Mountain Car, the method corrects KDE model bias and achieves near-nominal 90% coverage (details in supplement).

Measured effects reported in the paper

Limitations

• The choice of rollout horizon k is underexplored; multiple k give correlated intervals that are hard to aggregate.
• Coverage guarantees are asymptotic (n to infinity) and depend on importance-weight estimation error and the bounded-density assumption.
• Evaluation limited to simple synthetic environments and Mountain Car; no broader benchmarks or policy optimization.

Experimental setup

Datasets: synthetic two-state MDP synthetic continuous-state MDP Mountain Car

Benchmarks: synthetic two-state MDP synthetic continuous-state MDP Mountain Car

Metrics: coverage probability average interval length

Baselines: DRL-QR KDE-QR

Builds on: split conformal prediction quantile temporal difference learning weighted conformal prediction k-step temporal difference learning

Closest papers by meaning

• Beyond Expected Return: Accounting for Policy Reproducibility When Evaluating Reinforcement Learning Algorithms (AAAI 2024 · similarity 0.90)
• Selective Uncertainty Propagation in Offline RL (AAAI 2025 · similarity 0.88)
• Dynamic Uncertainty Estimation for Offline Reinforcement Learning (AAAI 2025 · similarity 0.87)
• Intrinsic Benefits of Categorical Distributional Loss: Uncertainty-aware Regularized Exploration in Reinforcement Learning (NeurIPS 2025 · similarity 0.87)
• Reinforcement Learning with Imperfect Transition Predictions: A Bellman-Jensen Approach (NeurIPS 2025 · similarity 0.87)
• Deterministic Uncertainty Propagation for Improved Model-Based Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.87)