Research map / Reinforcement learning / Policy gradient and actor-critic
Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives fully empirical high-probability bounds for regularized IPS estimators in offline contextual bandits and proposes Logarithmic Smoothing (LS), a log-based IW-smoothing estimator whose bound is tighter than prior bounds, applied to pessimistic OPE, OPS and OPL.
Problem
Off-policy evaluation, selection and learning in offline contextual bandits require risk estimates with reliable worst-case guarantees, but point estimators like IPS have high variance and existing pessimistic bounds are tied to predefined estimators.
Approach
Derive general empirical moment-based concentration bounds for a broad class of regularized IPS estimators and select the regularizer minimizing the bound as L→∞, which yields Logarithmic Smoothing: -1/λ log(1-λ w c), a soft differentiable clipping that is provably tighter than IX and clipping bounds.
Key results
- On kropt with 2^14 samples, LS bound relative radius 0.035 vs 0.041 for IX and 0.257 for SN-ES.
- In OPL, LS-LIN achieves guaranteed-risk relative improvement 26.31% vs 24.74% for IX and 14.48% for cIPS.
- In OPS over 110 scenarios, the LS pessimistic selection never picks policies worse than pi_0 while finding the best policy often (comparable to SN).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Logarithmic Smoothing instead of the strongest competing risk bound | policy risk evaluation on kropt · kropt, 2^8–2^14 samples | relative radius |U(π)/R(π) - 1|: 0.364 -> 0.362; 0.257 -> 0.236; 0.226 -> 0.213; 0.171 -> 0.159; 0.126 -> 0.117; 0.062 -> 0.054; 0.041 -> 0.035 | Helped |
| Use LS-LIN instead of the strongest competing PAC-Bayesian bound | PAC-Bayesian off-policy learning · OPL, 200 scenarios | rI(U(π̂ₙˡ)): 24.74% -> 26.31% | Helped |
| Use LS-LIN instead of the strongest competing PAC-Bayesian bound | PAC-Bayesian off-policy learning · OPL, 200 scenarios | rI(R(π̂ₙˡ)): 36.70% -> 36.76% | No clear effect |
| Use pessimistic selection with the LS upper bound instead of point-estimator selection | off-policy policy selection · 11 UCI datasets, 110 scenarios | selection of poor-performing policies | Helped |
| Increase empirical moment order from L=1 to L>1 using the optimal regularizer (ablation) | risk upper bound · offline contextual bandit risk evaluation | upper-bound tightness | Helped |
| Use Logarithmic Smoothing rather than the IX bound | policy risk evaluation · offline contextual bandit | upper-bound tightness | Helped |
| Use logarithmic smoothing (λ > 0) instead of IPS (λ = 0) | importance-weighted risk estimation · offline contextual bandit | variance: λ = 0: E[w²(x,a)c²]/n; λ > 0: 1/λn | Helped |
| Use clipping with the L=1 empirical moment bound instead of the empirical Bernstein bound | risk upper-bound evaluation · offline contextual bandit | concentration tightness | Helped |
| Use LS-LIN instead of the strongest competing bound | PAC-Bayesian off-policy learning · OPL, 200 scenarios | rI(R(π̂ₙˡ)): 36.70% -> 36.76% | No clear effect |
Surprising findings
- The optimal regularizer for the first-moment (L=1) bound is clipping, but as L grows to infinity the optimal regularizer becomes plain IPS, which is then smoothed by the log transform.
Limitations
- Performs poorly in extremely large action spaces due to importance weighting limitations.
- Assumes i.i.d. contextual bandit data, not applicable to long-term effects or full RL.
- Bounds only apply to importance-weighting estimators; doubly robust extension is nontrivial and requires negative costs.
Experimental setup
Datasets: kropt balance-scale UCI ML Repository multiclass datasets (11 for OPE/OPS, 4 for OPL)
Benchmarks: kropt 11 UCI multiclass datasets converted to bandit problems 4 UCI datasets for OPL
Metrics: relative radius of bound |U/R-1| fraction of best/better/worse policy selected relative improvement rI of guaranteed and true risk
Baselines: SN-ES cIPS-EB IX IPS SN cIPS cvcIPS ES
Builds on: inverse propensity scoring implicit exploration clipping exponential smoothing PAC-Bayes off-policy learning
Closest papers by meaning
- Empirical Likelihood for Contextual Bandits (NeurIPS 2020 · similarity 0.91)
- Log-Sum-Exponential Estimator for Off-Policy Evaluation and Learning (ICML 2025 · similarity 0.90)
- Conformal Off-Policy Prediction in Contextual Bandits (NeurIPS 2022 · similarity 0.90)
- PAC-Bayesian Offline Contextual Bandits With Guarantees (ICML 2023 · similarity 0.90)
- Doubly robust off-policy evaluation with shrinkage (ICML 2020 · similarity 0.90)
- Off-Policy Risk Assessment in Contextual Bandits (NeurIPS 2021 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.