Research map / Reinforcement learning / Policy gradient and actor-critic
Loaded DiCE: Trading off Bias and Variance in Any-Order Score Function Gradient Estimators for Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use an exact value function in the advantage estimator | Loaded DiCE on Random MDPs · Random MDPs | derivative estimator variance | Helped |
| Use an advantage estimator rather than sampled costs alone | Loaded DiCE on Random MDPs · Random MDPs | derivative convergence | Helped |
| Use LVC instead of Loaded DiCE | higher-order derivative estimation · Random MDPs | derivative bias | Hurt |
| Lower τ in the advantage estimator (ablation) | Loaded DiCE with an approximate value function · Random MDPs | derivative variance | Helped |
| Lower τ in the advantage estimator (ablation) | Loaded DiCE with an approximate value function · Random MDPs | derivative bias | Hurt |
| Discount more distant causal dependencies with lower λ (ablation) | Loaded DiCE · Random MDPs | higher-order derivative variance | Helped |
| Discount more distant causal dependencies with lower λ (ablation) | Loaded DiCE · Random MDPs | higher-order derivative bias | Hurt |
| Vary λ (ablation) | Loaded DiCE · Random MDPs | first-order derivative estimates | No clear effect |
| Use λ=1.0 rather than a lower λ (ablation) | Loaded DiCE inner-loop estimator · CheetahDir and CheetahVel | post-adaptation return | Hurt |
| Use a low but non-zero λ (ablation) | Loaded DiCE inner-loop estimator · CheetahVel | learning speed | Helped |
| Use a non-zero λ instead of λ=0 (ablation) | Loaded DiCE inner-loop estimator · CheetahVel | final post-adaptation return | No clear effect |
| Use too high a τ (ablation) | Loaded DiCE inner-loop estimator · CheetahDir | post-adaptation return | Hurt |
Closest papers by meaning
- A Baseline for Any Order Gradient Estimation in Stochastic Computation Graphs (ICML 2019 · similarity 0.92)
- DiCE: The Infinitely Differentiable Monte Carlo Estimator (ICML 2018 · similarity 0.90)
- Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluation (NeurIPS 2021 · similarity 0.90)
- ODICE: Revealing the Mystery of Distribution Correction Estimation via Orthogonal-gradient Update (ICLR 2024 · similarity 0.88)
- Exclusively Penalized Q-learning for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
- Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.