Research map / Time series / Causal and generative time series

ERCI: An Explainable Experience Replay Approach with Causal Inference for Deep Reinforcement Learning

AAAI 2025 · Time series › Causal and generative time series · approach: Dynamical system reconstruction · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ERCI, an experience replay method that extracts time series causal factors from DRL action trajectories, runs GFCI causal discovery to score causal strength, and uses those scores as sampling weights, reporting gains over DDPG and TD3 in Highway-Env, Gym, and a CARLA overtaking case.

Problem

Standard experience replay in deep reinforcement learning samples transitions without capturing the internal causal structure of training data, which makes DRL training hard to explain and leaves sample efficiency unexploited, especially in safety-critical settings like autonomous driving.

Approach

Segments the action time series into non-overlapping explainable patterns (TSCFs) via two-stage clustering (TICC then K-Means with DTW), infers causal relations between these factors with GFCI on a PAG, and uses the resulting path-summed causal strengths as softmax-based sampling weights in a causal replay weight tree with curriculum decay.

Key results

• DDPG-ERCI improves AS over DDPG by 20.70% on average across environments (Table 2), with SAS reduced by 22.99%.
• TD3-ERCI improves AS over TD3 by 8.92% and BS by 14.96% (Table 2).
• DDPG-PERCI improves AS over DDPG-PER by 24.52% (Table 3); the paper's abstract claims a 15% average gain over state-of-the-art methods.

Measured effects reported in the paper

Limitations

• Causal strength is computed from binary TSCF presence variables and reward, so temporal ordering is only approximated by a heuristic for uncertain edges.
• Evaluation limited to DDPG/TD3 variants and a few environments, with results reported only for the first 1000 episodes.
• The paper reports the average-score gain for DDPG as 20.70% and abstract reports 15%, and the SAS improvement for PERCI is small.

Experimental setup

Datasets: Highway-Env (Highway, Intersection, Racetrack) Gym Reacher Gym Humanoid CARLA overtaking scenario (300 instances)

Benchmarks: Highway-Env Highway Highway-Env Intersection Highway-Env Racetrack Gym Reacher Gym Humanoid

Metrics: Average Score (AS) Best Score (BS) Step of Average Score (SAS) Average Cumulative Score (ACS)

Baselines: DDPG TD3 DDPG-PER TD3-PER

Builds on: DDPG TD3 Prioritized Experience Replay GFCI TICC

Closest papers by meaning

• Explainable Reinforcement Learning through a Causal Lens (AAAI 2020 · similarity 0.87)
• Remember and Forget for Experience Replay (ICML 2019 · similarity 0.87)
• Revisiting Fundamentals of Experience Replay (ICML 2020 · similarity 0.87)
• CausalXRL: Explainable Reinforcement Learning through Causal Graph Reasoning (ICML 2026 · similarity 0.87)
• Turning Sand to Gold: Recycling Data to Bridge On-Policy and Off-Policy Learning via Causal Bound (NeurIPS 2025 · similarity 0.86)
• Generalization of RLVR Using Causal Reasoning as a Testbed (ICLR 2026 · similarity 0.85)