Research map / Reinforcement learning / Imitation learning
Dual RL: Unification and New Methods for Reinforcement and Imitation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Unifies recent offline RL and imitation learning methods as instances of "dual RL" (Bellman-flow LP duals), then proposes discriminator-free ReCOIL for offline IL from arbitrary data and f-DVL, a family of stable implicit-maximizer alternatives to XQL's Gumbel loss.
Problem
Offline RL and imitation learning methods have been developed under disparate derivations, and prior offline IL methods need coverage assumptions and discriminators, while XQL for offline RL suffers from training instability due to its Gumbel (exponential) loss.
Approach
Cast many offline RL/IL algorithms (IQLearn, XQL, CQL, OPOLO, SMODICE, etc.) as instances of unconstrained Lagrangian dual problems over state-action visitation (dual-Q / dual-V); from this, match mixture distributions of expert and suboptimal data to get a discriminator-free IL objective (ReCOIL) and replace the Gumbel loss with polynomial conjugates (chi^2, TV) for stable implicit value maximiza
Key results
- ReCOIL reaches 108.18 on hopper random+expert with 1 expert demo versus 73.93 for ORIL and 101.61 for SMODICE, and substantially beats prior IL methods on most manipulation tasks (e.g., pen cloned+expert 95.04 vs 34.94 BC full).
- f-DVL (chi^2 / TV) matches or beats XQL on D4RL offline RL in many tasks, e.g., antmaze-medium-play 71.0 (TV) vs 31.2 for XQL(r) and hopper-medium-replay 98.0 (TV) vs 95.1.
- ReCOIL recovers reward with Pearson correlation 0.98 on Hopper and 0.92 on Walker.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use discriminator-free ReCOIL with arbitrary off-policy data instead of coverage-dependent offline IL | Offline imitation learning · D4RL locomotion and manipulation tasks | normalized return: 108.18$\pm$3.28 vs 101.61$\pm$7.69; 95.04$\pm$4.48 vs 34.94$\pm$11.10 | Helped |
| Use total variation divergence in f-DVL instead of the reverse-KL Gumbel loss used by XQL | Offline RL · D4RL tasks | training stability | Helped |
| Use f-DVL with Pearson chi-square or total variation divergence instead of XQL(r) | Offline RL · D4RL continuous-control tasks | normalized return: hopper-medium-v2: 68.5 vs 63.0; hopper-medium-replay-v2: 95.1 vs 98.0 | Mixed |
| Use ReCOIL to estimate policy visitation rather than coverage-dependent SMODICE or expert-only IQLearn | Offline imitation learning · Policy visitation estimation | MSE | Helped |
| Use ReCOIL to recover the reward function | ReCOIL · Hopper and Walker | Pearson correlation: 0.98; 0.92 | Helped |
Surprising findings
- The reported XQL results are not reproducible under the standard last-iterate evaluation protocol; the authors' reproduced XQL(r) scores are much lower on several tasks (e.g., antmaze-umaze 47.7 vs reported higher).
- SMODICE performs poorly when suboptimal data has few expert samples (e.g., random+few-expert hopper 60.11 vs ReCOIL 97.85).
Limitations
- Framework restricted to regularized RL with alpha > 0 and static regularization distribution; does not cover supervised methods like DT or RvS.
- Assumes expert demonstrations are high quality.
- Experiments focus on the offline setting; online extension is only partly evaluated.
Experimental setup
Datasets: D4RL D4RL random D4RL medium D4RL medium-replay D4RL medium-expert D4RL human D4RL cloned D4RL partial D4RL mixed
Benchmarks: D4RL MuJoCo locomotion Adroit manipulation Franka Kitchen
Metrics: normalized return MSE of visitation estimate Pearson correlation of recovered reward
Baselines: RCE ORIL SMODICE IQLearn BC XQL CQL IQL TD3+BC DT
Builds on: IQLearn XQL SMODICE CQL DICE
Closest papers by meaning
- An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learning (ICLR 2025 · similarity 0.93)
- Hybrid Inverse Reinforcement Learning (ICML 2024 · similarity 0.91)
- Is Value Learning Really the Main Bottleneck in Offline RL? (NeurIPS 2024 · similarity 0.91)
- Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective (ICML 2024 · similarity 0.90)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- RLIF: Interactive Imitation Learning as Reinforcement Learning (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.