Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Explaining Reinforcement Learning Agents through Counterfactual Action Outcomes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes COViz, a local XRL method that visualizes counterfactual (second-best action) trajectories against the agent's chosen trajectory, and tests it alone and combined with reward decomposition in two user studies on highway driving agents.
Problem
Local explanations in explainable RL typically explain why an action was chosen but do not show the outcomes of the chosen action versus alternatives, making it hard for laypeople to understand agent preferences.
Approach
Generate counterfactual trajectories by forcing the agent's second-best action at each state and rolling out its policy for k steps, then display them side-by-side with the true trajectory; adapt DISAGREEMENTS to a single agent and combine the visualization with reward decomposition bars.
Key results
- In study 1 (N=115), the integrated CORD condition reached mean correctness 0.64 versus 0.45 for RD and 0.42 for COViz (both p<0.001).
- In study 2 (N=49, within-subject), CORD reached 0.63 versus 0.48 for RD and 0.40 for COViz, with significant differences.
- Despite better performance, CORD was ranked last by 53% of participants in study 2, while COViz was ranked first by 57%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Combine COViz with reward decomposition instead of using either explanation alone | Agent preference explanations · Study 1 | correctness rate: RD: M=0.45 -> CORD: M= 0.64; COViz: M=0.42 -> CORD: M= 0.64 | Helped |
| Combine COViz with reward decomposition instead of using either explanation alone | Agent preference explanations · Study 2 | correctness rate: RD: M=0.48 -> CORD: M= 0.63; COViz: M= 0.4 -> CORD: M= 0.63 | Helped |
| Use COViz rather than reward decomposition alone | Agent preference explanations · Study 1 | correctness rate: COViz: M=0.42; RD: M=0.45; p = 0.22 | No clear effect |
| Use COViz rather than reward decomposition alone | Agent preference explanations · Study 2 | correctness rate: COViz: M= 0.4; RD: M=0.48; p = 0.064 | No clear effect |
| Add COViz and reward decomposition to summary-only explanations | Agent preference explanations · Study 1 | correctness rate: SUM: M= 0.31; CORD: M= 0.64; U=607, p=0.002 | Helped |
| Select states using Q-value differences between actions | COViz global explanations · Highway environment | counterfactual trajectory divergence | Hurt |
| Select states by sampling execution traces according to state frequency | Local explanations · Frequency-based state selection | correctness rate | Mixed |
Surprising findings
- The integrated explanation gave the best objective performance yet was the least preferred by participants.
- Q-value-based state importance selected states where the counterfactual trajectory nearly reverted to the original path, making counterfactuals almost identical to true trajectories.
Limitations
- Only a single counterfactual action deviation is forced, which may be reversible and negligible in continuous domains.
- Single domain (highway) evaluated.
- Combined explanation imposes higher cognitive load and was least preferred by participants.
Experimental setup
Datasets: highway environment simulation traces
Benchmarks: user study on identifying agent reward-component preferences (highway)
Metrics: fraction of correct reward component comparisons explanation satisfaction (7-point Likert) confidence (5-point Likert) explanation ranking
Baselines: Reward Decomposition Policy summary (SUM) COViz alone
Builds on: DISAGREEMENTS HIGHLIGHTS Hierarchical Reward Architecture DDQN
Closest papers by meaning
- What Did You Think Would Happen? Explaining Agent Behaviour through Intended Outcomes (NeurIPS 2020 · similarity 0.91)
- Explanations for Multi-Agent Reinforcement Learning (AAAI 2025 · similarity 0.87)
- Understanding Individual Agent Importance in Multi-Agent System via Counterfactual Reasoning (AAAI 2025 · similarity 0.87)
- Approximating Shapley Explanations in Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- ExPO: Unlocking Hard Reasoning with Self-Explanation-Guided Reinforcement Learning (NeurIPS 2025 · similarity 0.86)
- Explaining Decisions of Agents in Mixed-Motive Games (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.