Research map / Reinforcement learning / Policy gradient and actor-critic
Distributional Off-Policy Evaluation for Slate Recommendations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SUnO, an unbiased, consistent off-policy estimator of the full reward CDF for slate recommendations, using sum-of-slot importance weights under an additive CDF assumption.
Problem
Off-policy evaluation of slate recommendation policies typically targets only expected reward; estimating the full reward distribution (for risk metrics like VaR/CVaR) is intractable with product importance weights in the combinatorial slate action space.
Approach
Replace the product of slot-level importance ratios (as in UnO) with a sum of slot-level density ratios, G = sum_k(pi/mu_k - 1) + 1, which remains unbiased when the slate reward CDF decomposes additively over slots, giving variance growing linearly rather than exponentially in slate size K.
Key results
- On the MovieLens simulator, SUnO's average KS statistic was 0.169-0.184 versus UnO's 0.441-0.252 across 0.5M-10M samples.
- On the synthetic experiment, SUnO's KS statistic was 0.131 at 500 samples versus UnO's 0.256, and 0.049 vs 0.077 at 10k.
- In the non-additive Open Bandit Pipeline setting, SUnO's mean-estimation MSE was about 0.25-0.27 versus UnO's 0.54-0.57.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SUnO's sum of slot-level density ratios instead of UnO's product importance weight | Slate distribution off-policy evaluation · Synthetic experiment | Kolmogorov-Smirnov statistic: 0.256 -> 0.131; 0.191 -> 0.102; 0.098 -> 0.059; 0.077 -> 0.049 | Helped |
| Use SUnO's sum of slot-level density ratios instead of UnO's product importance weight | Slate distribution off-policy evaluation · MovieLens-20M | Kolmogorov-Smirnov statistic: 0.441 -> 0.169; 0.410 -> 0.173; 0.31912 -> 0.181; 0.252 -> 0.184 | Helped |
| Use SUnO's structured estimator instead of the structure-agnostic UnO estimator | Slate mean estimation · Open Bandit Pipeline | Mean squared error: 0.543 -> 0.253; 0.541 -> 0.257; 0.567 -> 0.269 | Helped |
Surprising findings
- SUnO still outperforms UnO in the non-additive OBP setting even though its unbiasedness guarantee does not hold there.
Limitations
- Unbiasedness requires the additive CDF assumption, which is not verified on real data and cannot be guaranteed in general.
- Risk metrics such as CVaR and variance are non-linear in the CDF, so their estimates are biased.
- Experiments use small slates and simulators (K=3 or 5), with no variance-reduction extensions such as self-normalization or control variates.
Experimental setup
Datasets: MovieLens-20M Open Bandit Pipeline synthetic slate simulator
Benchmarks: MovieLens-20M slate simulator synthetic slate simulator Open Bandit Pipeline cascade additive reward
Metrics: Kolmogorov-Smirnov statistic mean squared error VaR CVaR median estimation error
Baselines: UnO
Builds on: Universal off-policy evaluation (UnO) additive slate reward estimators inverse propensity scoring
Closest papers by meaning
- Off-policy evaluation for slate recommendation (NeurIPS 2017 · similarity 0.94)
- Control Variates for Slate Off-Policy Evaluation (NeurIPS 2021 · similarity 0.92)
- Off-Policy Evaluation for Large Action Spaces via Embeddings (ICML 2022 · similarity 0.87)
- Active Offline Policy Selection (NeurIPS 2021 · similarity 0.87)
- Uncertainty-Aware Instance Reweighting for Off-Policy Learning (NeurIPS 2023 · similarity 0.87)
- OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.