Research map / Reinforcement learning / Policy gradient and actor-critic
Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search
Lars Buesing, Theophane Weber, Yori Zwols, Nicolas Heess, Sebastien Racaniere, Arthur Guez, Jean-Baptiste Lespiau
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Anchor model rollouts in logged episodes by inferring scenarios counterfactually | CF-GPS · PO-SOKOBAN | policy performance | Improved |
| Replace counterfactual scenario inference with an unconditional initial-state model (ablation) | CF-GPS policy search · PO-SOKOBAN | policy performance | Hurt |
| Condition scenario inference only on the first observation instead of hindsight data (ablation) | CF-GPS policy search · PO-SOKOBAN | policy performance | Hurt |
| Increase logged episode information used to infer initial scenarios (ablation) | CF-PE · PO-SOKOBAN policy evaluation | policy evaluation error | Improved |
| Use no real-data conditioning for scenario inference (ablation) | CF-PE · PO-SOKOBAN policy evaluation | policy evaluation error | Hurt |
| Use Importance Sampling rather than counterfactual inference | Off-policy policy evaluation · PO-SOKOBAN | relative error: >0.8 relative error | Hurt |
Closest papers by meaning
- Counterfactual Distillation: Internalizing Reflective Experience into LLM Agent (NeurIPS 2026 · similarity 0.89)
- Offline RL with Observation Histories: Analyzing and Improving Sample Complexity (ICLR 2024 · similarity 0.88)
- Compositional Conservatism: A Transductive Approach in Offline Reinforcement Learning (ICLR 2024 · similarity 0.88)
- Efficient Policy Evaluation with Offline Data Informed Behavior Policy Design (ICML 2024 · similarity 0.88)
- Abstract Reward Processes: Leveraging State Abstraction for Consistent Off-Policy Evaluation (NeurIPS 2024 · similarity 0.88)
- Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
