Research map / Reinforcement learning / Policy gradient and actor-critic
Surrogate Objectives for Batch Policy Optimization in One-step Decision Making
Minmin Chen, Ramki Gummadi, Chris Harris, Dale Schuurmans
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace direct expected-reward (R-hat) objective with reverse KL surrogate | Policy training, full feedback · CIFAR-10 full reward feedback | test error %: 19.9% -> 10.7% | Improved |
| Replace log-likelihood with reverse KL surrogate | Policy training, full feedback · CIFAR-10 full reward feedback | test error %: 10.4% -> 10.7% | No clear effect |
| Use composite surrogate instead of direct R-hat on random-behavior bandit data | Policy training, bandit feedback · MNIST and CIFAR-10 uniform behavior policy | test error | Improved |
| Use composite surrogate instead of direct R-hat on benchmark bandit data | CIFAR-10 bandit data of [14] · CIFAR-10 bandit feedback (50k to 250k) | test error %: 8.71 -> 8.34 (50k); 6.66 -> 6.36 (250k) | Improved |
| Add squared-error term and R-hat mix to reverse KL (composite vs reverse KL alone) (ablation) | Surrogate objective · CIFAR-10 bandit feedback (50k) | test error %: 16.00 -> 8.34 | Improved |
| Use composite surrogate instead of R-hat on Criteo | Linear policy on Criteo · Criteo test | importance-sampled reward x10^4: 51.72 ± 1.42 -> 55.09 ± 2.86 | Improved |
| Use reverse KL alone instead of R-hat on Criteo (ablation) | Linear policy on Criteo · Criteo test | importance-sampled reward x10^4: 51.72 ± 1.42 -> 52.30 ± 0.83 | No clear effect |
| Use squared error alone instead of log-likelihood or surrogate (ablation) | Policy training, full feedback · MNIST and CIFAR-10 | test error | Hurt |
Closest papers by meaning
- On the Optimality of Batch Policy Optimization Algorithms (ICML 2021 · similarity 0.90)
- Batched Thompson Sampling (NeurIPS 2021 · similarity 0.89)
- Active Context Selection Improves Simple Regret in Contextual Bandits (NeurIPS 2026 · similarity 0.88)
- Small steps no more: Global convergence of stochastic gradient bandits for arbitrary learning rates (NeurIPS 2024 · similarity 0.88)
- Batched Stochastic Linear Bandits with 1-Bit Communication Constraints (NeurIPS 2026 · similarity 0.87)
- Cost-Aware Near-Optimal Policy Learning (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
