Research map / Reinforcement learning / Policy gradient and actor-critic
Cross-Domain Off-Policy Evaluation and Learning for Contextual Bandits
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Cross-Domain OPE/L, using logged data from source domains to evaluate and learn policies in a target domain with deterministic logging, new actions, and few-shot data, via the COPE estimator and COPE-PG policy gradient.
Problem
Off-policy evaluation and learning in contextual bandits fail under few-shot target data, deterministic logging policies, and new actions because importance weighting needs common support and suffers high variance.
Approach
Decompose the expected reward into a domain-cluster effect, estimated unbiasedly via multiple importance weighting over source domains in the target's cluster, and a domain-specific effect, handled by reward regression on target data; extend to a policy gradient (COPE-PG).
Key results
- In OPE on KuaiRec with 1 data point per action (n^T/|A|=1), COPE achieves MSE ratio 6.79 relative to the best baseline DM(T).
- COPE(Ours) has the lowest MSE across all new-action ratios in the OPE table (e.g., 1.1641 vs. 2.1763 for IPS(ALL) at 0.0 new actions).
- COPE-PG achieves the highest relative policy value in most OPL settings, e.g., 1.4463 vs. 1.4308 for DR-PG(ALL) at 0.0 new actions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use COPE instead of the strongest listed baseline | KuaiRec OPE · New-action ratios 0.0–0.8 | MSE: 1.1641 vs 2.1763; 1.1647 vs 2.2401; 0.8066 vs 2.3985; 1.0663 vs 2.1881; 1.0801 vs 1.4530 | Helped |
| Use COPE instead of IPS-ALL | KuaiRec OPE · Deterministic logging in 200–1000 users | MSE: 1.3238 vs 2.1561; 1.3120 vs 2.1630; 1.2926 vs 2.1906; 1.2125 vs 2.2188; 1.2172 vs 2.2076 | Helped |
| Use COPE rather than DM with one data point per action | KuaiRec OPE · Target-domain data size n^T / |A| = 1 | MSE ratio (DM to COPE): 6.79 | Helped |
| Use COPE-PG instead of the strongest listed baseline | KuaiRec OPL · New-action ratios 0.0–0.6 | relative policy value: 1.4463 vs 1.4308; 1.4060 vs 1.3947; 1.4389 vs 1.4064; 1.3884 vs 1.3420 | Helped |
| Use the all-domain DR-PG baseline instead of COPE-PG | KuaiRec OPL · New-action ratio 0.8 | relative policy value: 1.4403 vs 1.4090 | Hurt |
| Use COPE-PG rather than target-only or all-domain policy-gradient baselines | KuaiRec OPL · Increasing deterministic logging users | relative policy value | Mixed |
| Vary the target-cluster size (ablation) | COPE-PG · KuaiRec OPL | relative policy value | Mixed |
| Use small rather than larger target clusters (ablation) | COPE · KuaiRec OPE | bias and variance | Mixed |
| Vary the user-distribution dependence parameter λ (ablation) | COPE-PG · KuaiRec OPL | relative policy value: 1.4713; 1.4646; 1.4739; 1.4502; 1.4780; 1.4404 | Helped |
Surprising findings
- IPS and DR MSE do not deteriorate rapidly as new actions increase, because variance drops as supported actions decrease.
Limitations
- CPC condition is unlikely to hold exactly in practice; bias depends on regression accuracy and cluster choice.
- Domain clustering is heuristic, based on empirical average rewards.
- Variance can increase with larger target clusters due to density ratio terms.
Experimental setup
Datasets: KuaiRec
Benchmarks: KuaiRec
Metrics: MSE squared bias variance test policy value relative to logging policy
Baselines: IPS(T) IPS-ALL DR(T) DR-ALL DM DM-ALL IPS-PG DR-PG
Builds on: doubly robust estimator OffCEM inverse propensity scoring uLSIF density ratio estimation
Closest papers by meaning
- Off-Policy Learning in Large Action Spaces: Optimization Matters More Than Estimation (ICML 2026 · similarity 0.90)
- Optimal and Adaptive Off-policy Evaluation in Contextual Bandits (ICML 2017 · similarity 0.90)
- A General Framework for Off-Policy Learning with Partially-Observed Reward (ICLR 2025 · similarity 0.90)
- Budgeted Multi-Source Counterfactual Annotation for Off-Policy Evaluation (NeurIPS 2026 · similarity 0.89)
- Log-Sum-Exponential Estimator for Off-Policy Evaluation and Learning (ICML 2025 · similarity 0.88)
- OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.