Research map / Reinforcement learning / Visual and representation RL

ACAMDA: Improving Data Efficiency in Reinforcement Learning through Guided Counterfactual Data Augmentation

AAAI 2024 · Reinforcement learning › Visual and representation RL · approach: Scalable RL systems and environments · 7 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ACAMDA, which recovers temporal causal structure from heterogeneous non-expert trajectories and uses a bidirectional conditional causal GAN (BiCCGAN) to generate guided counterfactual transitions that augment DQN policy training.

Problem

Reinforcement learning is data-inefficient, and counterfactual data augmentation in heterogeneous environments often relies on correlations rather than causal structure and suffers from model bias and unstable GAN training.

Approach

Combine sparsity-penalized causal structure recovery with a bidirectional conditional GAN that takes causally relevant parents plus a domain-change factor λ, adds a bias factor L2 loss for stability, and generates counterfactual transitions via Pearl's abduction-action-prediction to augment the dataset for DQN.

Key results

• On the synthetic dataset, ACAMDA's counterfactual states reach PCC 0.97 with the ground truth versus 0.66 for an MLP-based method.
• ACAMDA reduces source samples relative to V-DQN by 36x on CP-Noisy SD, 11x on CP-Noisy HD, 34x on CP-Windy SD, 10x on CP-Windy HD, and 8x on inventory.
• In CP-Windy HD ACAMDA reaches 196±2 successes versus at most 104±47 for other baselines, and in the inventory task it achieves AAR 301.362 versus 258.289 for P-MLP.

Measured effects reported in the paper

Surprising findings

• Simply adding the bias factor loss can underestimate dynamics in new domains without causal structure, per the authors' ablation.

Limitations

• Relies on the causal sufficiency assumption (no latent confounders).
• Evaluated only on small synthetic, CartPole, and inventory tasks rather than standard high-dimensional RL benchmarks.
• Counterfactual interventions use randomly selected alternative actions.

Experimental setup

Datasets: Synthetic VAR(1) dataset CartPole-Noisy (SD and HD) CartPole-Windy (SD and HD) Inventory task

Benchmarks: Synthetic VAR dataset CartPole-Noisy CartPole-Windy Inventory task

Metrics: Structural Hamming Distance Mean Squared Error Pearson Correlation Coefficient Success number Average Accumulative Reward

Baselines: V-DQN D-MLP P-MLP Conditional GAN BiCoGAN CoGAN

Builds on: CTRL BiCoGAN DQN CoDA MoCoDA

Closest papers by meaning

• Understanding when Dynamics-Invariant Data Augmentations Benefit Model-free Reinforcement Learning Updates (ICLR 2024 · similarity 0.88)
• Causal Action Influence Aware Counterfactual Data Augmentation (ICML 2024 · similarity 0.88)
• Causal Information Prioritization for Efficient Reinforcement Learning (ICLR 2025 · similarity 0.87)
• Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.86)
• Improving Generalization in Offline Reinforcement Learning via Adversarial Data Splitting (ICML 2024 · similarity 0.86)
• A Snapshot of Influence: A Local Data Attribution Framework for Online Reinforcement Learning (NeurIPS 2025 · similarity 0.85)