Research map / Reinforcement learning / Offline RL
BECAUSE: Bilinear Causal Representation for Generalizable Offline Model-based Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BECAUSE, an offline model-based RL method that learns a sparse bilinear causal representation of transitions under action-state confounding and uses an EBM-based pessimistic planner to mitigate objective mismatch.
Problem
Offline model-based RL suffers from objective mismatch between world-model learning and control, worsened by confounders and distribution shift between behavior data and online (possibly shifted) environments.
Approach
Models transition dynamics as a bilinear MDP with a sparse core matrix estimated as a causal mask via l0-regularized regression (with CIT for structure), reweights the mask by behavior policy to deconfound action-state confounders, and uses an energy-based uncertainty penalty in planning.
Key results
- BECAUSE achieves best performance in 8 of 9 settings across environments and data qualities for in-distribution and out-of-distribution evaluation.
- In ablation, BECAUSE averages 73.3% in-distribution vs 64.4% for the optimism variant and 39.3% for the full-batch variant (over 9 I and 9 O tasks; the reported ablation table is partly garbled).
- BECAUSE consistently outperforms MOPO across sample ratios and up to 8x confounder levels, with the margin growing as spurious level increases.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BECAUSE instead of the evaluated baselines | Offline RL agents · 18 grid world, manipulation, and autonomous driving tasks | success rate | Helped |
| Use BECAUSE instead of MOPO | Offline RL agents · Tasks with different offline-data sample ratios | success rate | Helped |
| Use BECAUSE instead of MOPO | Offline RL agents · Tasks with increased spurious levels | success rate | Helped |
| Use BECAUSE instead of ICIL and IFactor | Visual RL agents · Unlock visual tasks, in-distribution and out-of-distribution | success rate | Helped |
| Use causal representations in BECAUSE instead of MOPO | Offline model-based RL · Unlock, random demonstrations | model-control objective alignment | Helped |
Surprising findings
- Model-based approaches generally outperform model-free approaches across offline data levels.
Limitations
- Assumes time-homogeneous causal structure, which may fail in long-horizon or non-stationary settings.
- Implementation is limited to vector observations; visual scaling is only tested on a small Unlock setting.
Experimental setup
Datasets: offline data collected in Lift (RoboSuite) Unlock (Minigrid) Crash (highway-env)
Benchmarks: Lift Unlock Crash Lift-O Unlock-O Crash-O
Metrics: success rate mean and standard deviation over 10 seeds
Baselines: ICIL CCIL MnM Delphic TD3+BC MOPO GNN CDL Denoised MDP IFactor
Builds on: bilinear MDP MOPO differentiable causal discovery energy-based models
Closest papers by meaning
- Reining Generalization in Offline Reinforcement Learning via Representation Distinction (NeurIPS 2023 · similarity 0.90)
- Offline RL with Discrete Proxy Representations for Generalizability in POMDPs (NeurIPS 2023 · similarity 0.89)
- Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learning (ICML 2025 · similarity 0.88)
- Offline RL with Observation Histories: Analyzing and Improving Sample Complexity (ICLR 2024 · similarity 0.88)
- Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning (AAAI 2024 · similarity 0.87)
- Is Value Learning Really the Main Bottleneck in Offline RL? (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.