Research map / Reinforcement learning / Policy gradient and actor-critic
Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Measure action contributions toward rewarding outcomes instead of future states | COCOA vs. HCA+ · Linear key-to-door, key-door distance 100 | fraction of treasure rewards collected | Helped |
| Use rewarding outcomes rather than future states to estimate contributions | COCOA vs. HCA+ · Linear key-to-door, key-door distance 100 | signal-to-noise ratio | Helped |
| Use reward identity as the rewarding-outcome encoding | COCOA-reward · Linear key-to-door with increasing key-door distance | fraction of treasure rewards collected | Helped |
| Use learned features of rewarding objects as the rewarding-outcome encoding | COCOA-feature · Linear key-to-door with reward aliasing | fraction of treasure rewards collected | Helped |
| Use reward identity instead of learned rewarding-object features (ablation) | COCOA · Linear key-to-door with reward aliasing | fraction of treasure rewards collected | Hurt |
| Use rewarding-outcome contributions to separate tasks instead of estimating the full return | COCOA vs. policy-gradient baselines · Task interleaving | fraction of correct choices | Helped |
| Measure contributions toward states rather than rewarding outcomes (ablation) | HCA+ · Environments where action sequences lead deterministically to different states | policy-gradient variance | Hurt |
| Increase information content of the rewarding-outcome encoding (ablation) | COCOA · Tree environment | policy-gradient variance | Hurt |
| Increase overlap between neighboring nodes' children (ablation) | HCA+ · Tree environment | policy-gradient variance | Helped |
| Learn contribution coefficients rather than use ground-truth coefficients (ablation) | COCOA · Linear key-to-door | sample efficiency | Helped |
| Learn the full action-value function instead of contribution coefficients | Q-critic · Linear key-to-door | sample efficiency | Hurt |
| Use return-conditioned hindsight as an action-dependent baseline | HCA-return · Linear key-to-door | policy performance | Hurt |
Closest papers by meaning
- Latent Reward: LLM-Empowered Credit Assignment in Episodic Reinforcement Learning (AAAI 2025 · similarity 0.89)
- STAS: Spatial-Temporal Return Decomposition for Solving Sparse Rewards Problems in Multi-agent Reinforcement Learning (AAAI 2024 · similarity 0.87)
- Reducing Credit Assignment Variance via Counterfactual Reasoning Paths (NeurIPS 2026 · similarity 0.87)
- From Past to Future: Rethinking Eligibility Traces (AAAI 2024 · similarity 0.87)
- Sequence Compression Speeds Up Credit Assignment in Reinforcement Learning (ICML 2024 · similarity 0.87)
- Hindsight PRIORs for Reward Learning from Human Preferences (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.