Research map / Reinforcement learning / Visual and representation RL
Approximating Shapley Explanations in Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FastSVERL, a parametric amortised framework for approximating Shapley explanations (behaviour, outcome, prediction) in reinforcement learning, extended to off-policy and continual settings, plus a sampling-based variant that removes characteristic models.
Problem
Exact Shapley-value explanations of reinforcement learning agents scale exponentially with the number of state features and require expensive expectations over the state space, and must also handle temporal returns, off-policy data, and evolving policies.
Approach
Extends FastSHAP-style amortised Shapley models to RL by learning parametric characteristic models (behaviour, prediction, and a conditioned-policy value function for outcomes), uses importance sampling for passive off-policy learning, and proposes single-sample characteristic targets that remove the need for a separate characteristic model.
Key results
- Characteristic and Shapley models for behaviour, outcome, and prediction converge to low approximation error (<0.01) in Mastermind-222, with off-policy outcome training converging faster than on-policy.
- Training updates to reach target loss 0.01 scale roughly polynomially with number of states and are relatively insensitive to number of features in Hypercube.
- Single-sample characteristic targets matched exact-value performance while halving total training time and eliminating characteristic-model error propagation in Mastermind-222.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use off-policy rather than on-policy training for the outcome characteristic | FastSVERL outcome characteristic model · Mastermind-222 | training updates to convergence | Helped |
| Use an approximated rather than exact upstream characteristic | FastSVERL outcome Shapley model · Mastermind-222 | approximation error | Hurt |
| Apply importance sampling to off-policy training-buffer samples | FastSVERL behaviour characteristic model · Mastermind-222 | approximation error | Helped |
| Use importance sampling on off-policy data instead of on-policy data | FastSVERL behaviour characteristic model · Mastermind-222 | approximation error | Hurt |
| Increase explanation-model updates per policy update from 1:1 or 2:1 to 10:1 or 50:1 | FastSVERL prediction Shapley model during joint training · Mastermind-222 | approximation error | Helped |
| Replace the behaviour characteristic model with single-sample characteristic targets (ablation) | FastSVERL behaviour Shapley model · Mastermind-222 | training efficiency and approximation accuracy | Helped |
| Replace the behaviour characteristic model with single-sample characteristic targets rather than model-based targets (ablation) | FastSVERL behaviour Shapley model · Mastermind-222 | total training time | Helped |
| Use unnormalised importance sampling instead of no importance sampling (ablation) | FastSVERL characteristic models · Gridworld | approximation accuracy | Hurt |
Surprising findings
- Training with exact characteristic values converged slightly slower than with single-sample noisy targets, which the authors attribute to a stochastic regularisation effect.
- Off-policy training without importance sampling and with unnormalised importance weights degrade accuracy, and importance sampling does not match the on-policy baseline.
Limitations
- Outcome characteristic model has no convergence guarantee under deep RL with function approximation.
- Ground-truth accuracy cannot be validated in large domains; convergence loss is not a measure of accuracy.
- Experiments are on small, discrete-action gridworld-style domains; no user study; continuous state spaces require approximating the steady-state distribution.
Experimental setup
Datasets: Mastermind-222 Mastermind-333 Mastermind-443 Mastermind-453 Mastermind-463 Hypercube Gridworld
Benchmarks: Mastermind-222 Mastermind-333 Gridworld Hypercube Mastermind-443 Mastermind-453 Mastermind-463
Metrics: mean squared error vs exact values training updates to reach target loss final loss
Baselines: exact Shapley values model-based characteristic approximation Monte Carlo sampling
Builds on: SVERL FastSHAP DQN PPO
Closest papers by meaning
- Approximating the Shapley Value without Marginal Contributions (AAAI 2024 · similarity 0.87)
- Explaining Reinforcement Learning Agents through Counterfactual Action Outcomes (AAAI 2024 · similarity 0.86)
- Scalable Solutions for Decision-Making Systems Using Explainable Policy Representations (AAAI 2025 · similarity 0.85)
- ORCAID: Oblique Rule-Based Continuous-Action Interpretation for Deep RL Policies (NeurIPS 2026 · similarity 0.85)
- STAS: Spatial-Temporal Return Decomposition for Solving Sparse Rewards Problems in Multi-agent Reinforcement Learning (AAAI 2024 · similarity 0.84)
- Prompted Policy Search: Reinforcement Learning through Linguistic and Numerical Reasoning in LLMs (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.