Research map / Reinforcement learning / Visual and representation RL
Null Counterfactual Factor Interactions for Goal-Conditioned Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NCII, a null-counterfactual interaction inference method using a masked learned forward model, and HInt, which filters hindsight relabeling to trajectories with action-induced interactions, improving GCRL sample efficiency up to 4x.
Problem
Hindsight relabeling in goal-conditioned RL produces mismatched, trivial goals in object-centric domains where object-object interactions are rare, hurting sample efficiency; interactions lack a tractable statistical definition for GCRL.
Approach
Define interactions via a null counterfactual (a cause object is interacting with a target if removing it changes the target's transition distribution), infer them with a masked forward model and a learned interaction model, and keep only hindsight trajectories where the agent controls the target through an interaction path.
Key results
- NCII with Pointnet or Graph achieves lowest misprediction rate in most domains, e.g. 0.8% on 1-in Random DAG vs 1.6% for JACI and 4.5% for gradients.
- HInt improves GCRL sample efficiency by up to 4x over baselines across Spriteworld, Robosuite, Air Hockey, and Franka Kitchen.
- Interactions inferred by NCII yield HInt performance equivalent to or sometimes exceeding that with ground-truth interactions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use HInt interaction-filtered hindsight relabeling instead of vanilla hindsight | Goal-conditioned RL · Spriteworld, Robosuite, Air Hockey, and Franka Kitchen | sample efficiency: up to $4\times$ | Helped |
| Use NCII-inferred interactions rather than existing interaction inference methods | Interaction inference · Spriteworld, Air Hockey, Robosuite, and Kitchen | misprediction rate | Helped |
| Use NCII with Transformer backbone instead of Point or Graph backbone (ablation) | NCII · Random DAG Null | misprediction rate: 0.8 ± 0.2 -> 2.6 ± 0.3; 1.4± 0.2 -> 4.9 ± 1.7; 25.1± 5.8 -> 41.3 ± 8.2 | Hurt |
| Increase ε_null from 0.5 to 3.5 (ablation) | NCII · Sprite-1 | misprediction rate: 2.8±2.5 -> 43.5±17.9 | Hurt |
| Set ε_null to 0.7 rather than 0.5 (ablation) | NCII · 1-in | misprediction rate: 3.2±0.5 -> 1.1 ± 1.3 | Helped |
| Use NCII-inferred interactions instead of ground-truth interactions for HInt | HInt · Spriteworld default and small, Robosuite default, Air Hockey default, and Franka Kitchen | sample efficiency | No clear effect |
Surprising findings
- Transformer-based NCII backbone performs worse than PointNet or GNN on Random DAG domains.
Limitations
- Limited utility in domains where interactions are less critical, such as some locomotion tasks.
- Interactions can be detrimental in driving or drone navigation.
- Assumes a null state exists for each factor and that trajectories can contain varying subsets of factors, or relies on simulated nulling.
Experimental setup
Datasets: Random DAG Null Spriteworld Null Robosuite Robot Air Hockey Franka Kitchen
Benchmarks: Random DAG Null Spriteworld Null Robosuite Robot Air Hockey Franka Kitchen
Metrics: misprediction rate sample efficiency (training curves) success rate
Baselines: Vanilla DDPG Hindsight Prioritized Replay f-policy gradients ELDEN Causal Action Influence JACI Gradient Attention Neural Contextual Decomposition
Builds on: hindsight experience replay DDPG f-divergence policy gradients ELDEN Causal Action Influence
Closest papers by meaning
- How Does Goal Relabeling Improve Sample Efficiency? (ICML 2024 · similarity 0.84)
- Unsupervised Object Interaction Learning with Counterfactual Dynamics Models (AAAI 2024 · similarity 0.84)
- Stitching Sub-trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL (AAAI 2024 · similarity 0.84)
- PIPER: Primitive-Informed Preference-based Hierarchical Reinforcement Learning via Hindsight Relabeling (ICML 2024 · similarity 0.84)
- Provable Interactive Learning with Hindsight Instruction Feedback (ICML 2024 · similarity 0.83)
- Conservative Offline Goal-Conditioned Implicit V-Learning (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.