Research map / Reinforcement learning / Visual and representation RL

Explaining RL Decisions with Trajectories': A Reproducibility Study

NeurIPS 2024 · Reinforcement learning › Visual and representation RL · approach: In-context and adaptive RL

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

A reproducibility study of the trajectory-attribution XRL method ("Explaining RL decisions with trajectories"), partially confirming its claims on Grid-World and extending experiments to Seaquest, HalfCheetah, Breakout and Q*Bert.

Problem

Whether the claims of an offline RL explainability method that attributes agent decisions to clusters of training trajectories hold up under independent reproduction across environments.

Approach

Re-implements the trajectory-attribution pipeline (trajectory encoding, XMeans clustering, complementary datasets retraining policies, attribution frequency) for Grid-World, partially from authors' code and from scratch for other environments, then tests four claims with extra metrics, DBSCAN clustering, and alternative encoders.

Key results

• On Grid-World the original policy achieves the highest ISV (0.307), and the mean-cluster ISV (0.3029 reproduced vs 0.3027 original) closely matches the original paper.
• Seaquest reproduction differed substantially from the original paper's numbers, and only one of three high-level Seaquest behaviours was identified.
• Human study with 20 participants gave ~72.5% correct identification of one attributed trajectory and ~63% identifying all attributed trajectories, but the authors judged the evidence insufficient to confirm the claim.

Measured effects reported in the paper

Surprising findings

• Q*Bert highest ISV assigned to cluster 1, contradicting the claim that removing trajectories lowers ISV.
• Grid-World XMeans attribution yields no state explained solely by trajectories passing through it, with most attributed trajectories far from the state.

Limitations

• Original code only partially available (Grid-World); other environments implemented from scratch with unknown hyperparameters.
• Human study limited to 20 participants and Grid-World only.
• Only one cluster allowed per attribution, which the authors note as a limitation of the original method.

Experimental setup

Datasets: d4rl-Atari Seaquest Expert-offline RL Breakout and Q*Bert D4RL HalfCheetah Grid-World (generated with Dyna-Q)

Benchmarks: Grid-World 7x7 Grid-World Four Room 11x11 Seaquest HalfCheetah Breakout Q*Bert

Metrics: initial state value local mean absolute action-value difference action contrast measure Wasserstein distance cluster attribution frequency average distance of attributed trajectories

Baselines: DBSCAN BERT

Builds on: Explaining RL decisions with trajectories Dyna-Q XMeans DBSCAN SAC Decision Transformer

Closest papers by meaning

• Offline Preference-Based Trajectory Evaluation (NeurIPS 2026 · similarity 0.82)
• Decision Information Meets Large Language Models: The Future of Explainable Operations Research (ICLR 2025 · similarity 0.81)
• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs (ICLR 2026 · similarity 0.81)
• Think Twice, Act Once: A Co-Evolution Framework of LLM and RL for Large-Scale Decision Making (ICML 2025 · similarity 0.80)
• LARK: Learnability-Grounded Trajectory Selection for Efficient Reasoning Distillation (NeurIPS 2026 · similarity 0.80)
• Thought Branches: Interpreting LLM Reasoning Requires Resampling (ICLR 2026 · similarity 0.80)