Research map / Reinforcement learning / Offline RL

Causal Action Influence Aware Counterfactual Data Augmentation

ICML 2024 · Reinforcement learning › Offline RL · approach: Diffusion models for offline decision making · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CAIAC, an offline counterfactual data augmentation method that swaps action-unaffected entity states between trajectories, with action influence measured by state-conditioned conditional mutual information, to improve out-of-distribution robustness in goal-conditioned kitchen and Fetch manipulation tasks.

Problem

Offline learned robot policies suffer from causal confusion and spurious correlations in demonstrations, failing under distributional shift when the joint distribution over entities is poorly covered by data.

Approach

Detect, per state, which entities the agent's actions causally influence using point-wise conditional mutual information from a learned Gaussian transition model, then swap the action-uncontrollable entity transitions between independent trajectories to create feasible counterfactual transitions without environment interaction or counterfactual model rollouts.

Key results

• On the motivating kitchen experiment, CAIAC keeps OOD microwave task success equal to ID performance while CoDA and CoDA-action are drastically impaired and No-Augm. fails.
• On full Franka-Kitchen OOD evaluation, CAIAC achieves 0.81 on kettle and 0.75 on microwave success versus at most 0.18 for baselines, and is significantly better in 6 of 7 tasks.
• In low-data Fetch-Push (20%), CAIAC performs significantly better than no-augmentation, while CoDA, CoDA-action, and RSC degrade performance in all regimes.

Measured effects reported in the paper

Surprising findings

• Transformer attention-based causal influence estimates (CoDA) fail to recover the correct causal graph, even missing the diagonal self-influence, producing infeasible counterfactuals.
• Using only synthesized counterfactual data (ratio 1.0) degrades OOD performance, showing real data is needed to avoid selection bias.

Limitations

• Assumes sparse object-object interactions and that the agent is the only entity influencing others; may not hold in all tasks.
• Does not check counterfactual feasibility, so a small fraction of infeasible transitions are accepted.
• Relies on a learned CAI transition model and a hand-tuned influence threshold theta per environment.

Experimental setup

Datasets: D4RL kitchen (kitchen-complete, kitchen-partial, kitchen-mixed) Fetch-Push 2 cubes (self-collected) Fetch-Pick&Lift 4 cubes (self-collected)

Benchmarks: Franka-Kitchen (D4RL) OOD initial configurations Fetch-Pick&Lift 4 cubes Fetch-Push 2 cubes

Metrics: success rate log-likelihood under transition kernel ROC AUC for influence detection

Baselines: CoDA CoDA-action RSC No-Augm.

Builds on: Local Causal Model Causal Action Influence CoDA TD3+BC LMP

Closest papers by meaning

• Counterfactual Data Augmentation using Locally Factored Dynamics (NeurIPS 2020 · similarity 0.91)
• MoCoDA: Model-based Counterfactual Data Augmentation (NeurIPS 2022 · similarity 0.88)
• Offline Imitation Learning with Variational Counterfactual Reasoning (NeurIPS 2023 · similarity 0.88)
• ACAMDA: Improving Data Efficiency in Reinforcement Learning through Guided Counterfactual Data Augmentation (AAAI 2024 · similarity 0.88)
• Reward-Preserving Counterfactual State Editing for Offline Reinforcement Learning (ICML 2026 · similarity 0.86)
• Teaching Transformers Causal Reasoning through Axiomatic Training (ICML 2025 · similarity 0.86)