Research map / Reinforcement learning / Visual and representation RL
Composite Flow Matching for Reinforcement Learning with Shifted-Dynamics Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CompFlow, which models online transition dynamics as a flow composed on top of a pretrained offline flow, uses the flow-based Wasserstein distance as a dynamics-gap estimate for filtering offline data and driving optimistic exploration, and builds it on SAC for shifted-dynamics RL.
Problem
Offline data from a source environment with different transition dynamics can bias and destabilize online RL; existing dynamics-gap estimates based on KL or mutual information are ill-defined under mismatched support.
Approach
Composite flow: the online transition flow starts from the output distribution of a pretrained offline flow rather than a Gaussian, so the OT-based Wasserstein distance between offline and online transitions gives a well-defined, stable dynamics-gap estimate used both to filter offline data and to add an optimistic exploration bonus in high-gap regions.
Key results
- CompFlow achieves an average return of 2193 across 27 Gym-MuJoCo tasks, versus 1920 for the strongest baseline BC-SAC (about 14.2% relative improvement).
- CompFlow reports the best return on 19 of 27 tasks and ties on 5 more.
- On the wildlife patrol task, CompFlow outperforms the best baseline BC-PAR by 8.8% and training from scratch with SAC by 20.8%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CompFlow rather than the strongest baseline, BC-SAC | SAC with shifted-dynamics offline data · Gym-MuJoCo (27 tasks, average return) | return: 1920 -> 2193; 14.2% relative improvement | Helped |
| add composite flow matching, initializing the online flow from the offline flow rather than a Gaussian prior (ablation) | CompFlow transition model · Gym-MuJoCo transition validation set (held-out 10%) | MSE | Helped |
| set the offline data selection ratio ξ to 70 (ablation) | CompFlow · Walker Medium (Morphology) and Walker Medium Replay (Morphology) | return | Hurt |
| set the offline data selection ratio ξ to a moderate value (30 or 50) (ablation) | CompFlow · Gym-MuJoCo tasks | return | Helped |
| set the offline data selection ratio ξ to 20, filtering offline data more conservatively (ablation) | CompFlow · HalfCheetah tasks | return | Hurt |
| set the offline data selection ratio ξ to 70, admitting more offline data (ablation) | CompFlow · HalfCheetah tasks | return | Helped |
| add a larger exploration bonus β to target high-gap regions (ablation) | CompFlow · Friction tasks | return | Helped |
| use moderate rather than smaller or larger exploration strength β (ablation) | CompFlow · Morphology tasks | return | Helped |
| remove the exploration bonus (β=0) (ablation) | CompFlow · Five exploration-strength experiments | return | Hurt |
| use CompFlow rather than BC-PAR | SAC for wildlife conservation patrol policy learning · Queen Elizabeth National Park | reward: 8.8% | Helped |
| use CompFlow rather than training from scratch with SAC | SAC for wildlife conservation patrol policy learning · Queen Elizabeth National Park | reward: 20.8% | Helped |
Surprising findings
- Many recent off-dynamics baselines (H2O, BC-PAR, BC-VGDF) perform similarly to BC-SAC, suggesting limited ability to exploit offline data under dynamics shift.
Limitations
- Requires offline and online data to share state and action spaces.
- Evaluated only in simulation; no real-world experiments.
- Does not quantify uncertainty in the dynamics-gap estimate.
Experimental setup
Datasets: D4RL HalfCheetah medium D4RL HalfCheetah medium-replay D4RL HalfCheetah medium-expert D4RL Hopper medium D4RL Hopper medium-replay D4RL Hopper medium-expert D4RL Walker2d medium D4RL Walker2d medium-replay D4RL Walker2d medium-expert Murchison Falls wildlife incident data (simulator)
Benchmarks: Gym-MuJoCo HalfCheetah, Hopper, Walker2d with morphology, kinematic, friction shifts wildlife conservation patrol simulator
Metrics: average return after 40K environment interactions
Baselines: SAC BC-SAC H2O BC-VGDF BC-PAR
Builds on: Soft Actor-Critic flow matching optimal transport flow matching H2O BC-PAR
Closest papers by meaning
- PRECISE: SDE-Consistent Stochastic Sampling for RL Post-Training of Flow-Matching Models (NeurIPS 2026 · similarity 0.91)
- Designing Reinforcement Learning for Diffusion Models: A Unified Path-Space View (NeurIPS 2026 · similarity 0.90)
- Flow Matching for Offline Reinforcement Learning with Discrete Actions (NeurIPS 2026 · similarity 0.90)
- TMPO: Trajectory Matching Policy Optimization for Diverse and Efficient Diffusion Alignment (NeurIPS 2026 · similarity 0.89)
- Flow Matching Policy Optimization with Mirror Descent and Entropy Constraints (NeurIPS 2026 · similarity 0.89)
- Reward-Consistent Dynamics Models are Strongly Generalizable for Offline Reinforcement Learning (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.