Research map / Reinforcement learning / Offline RL

An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learning

ICLR 2025 · Reinforcement learning › Offline RL · approach: Offline RL sample complexity theory

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Iterative Dual-RL (IDRL), which corrects semi-gradient Dual-RL to recover the state-action visitation ratio and iteratively filters zero-weight transitions before discriminator-weighted behavior cloning, evaluated on D4RL and corrupted demonstrations.

Problem

Dual-RL offline methods using semi-gradient updates learn an action distribution ratio instead of the visitation ratio and are regularized toward the offline data distribution, limiting weighted-BC policy extraction performance.

Approach

Recover the state-action visitation ratio from the learned action ratio via an off-policy evaluation correction, then iteratively remove zero-weight transitions and rerun Dual-RL on the remaining subdataset, so the behavior distribution is progressively replaced by the optimized visitation distribution.

Key results

• IDRL reaches 90.8 average on Mujoco, 80.3 on Antmaze and 74.8 on Kitchen in the ablation table, with IDRL matching or beating baselines on most D4RL tasks (e.g., hopper-medium 99.5 vs. 86.1 for O-DICE).
• Using one iteration (M=1) drops Mujoco performance from 90.8 to 73.2, and filtering with the action ratio w(a|s) drops Mujoco to 56.8.
• IDRL uses 2 iterations of 10^6 steps each on D4RL and reports results across 7 seeds.

Measured effects reported in the paper

Surprising findings

• Semi-gradient Dual-RL (as in IQL/SQL) learns the action distribution ratio rather than the visitation ratio, which the paper argues hurts weighted-BC generalization.
• Discriminator-weighted BC trained with an expert dataset outperforms classical offline RL methods, motivating the approach.

Limitations

• Training time increases because multiple Dual-RL iterations are required.
• May suffer generalization issues when the dataset is small.
• Only one hyperparameter lambda, selected per dataset, though applied across tasks.

Experimental setup

Datasets: D4RL Mujoco (halfcheetah, hopper, walker2d medium/medium-replay/medium-expert) D4RL Antmaze D4RL Kitchen corrupted Mujoco mixtures of expert and random transitions

Benchmarks: D4RL Mujoco D4RL Antmaze D4RL Kitchen corrupted Mujoco demonstrations

Metrics: normalized score

Baselines: TD3+BC CQL ReBRAC Diffusion-QL X%-BC IQL SQL O-DICE Advantage-Weighted Density-Weighted (DW+AW)

Builds on: IQL SQL O-DICE DICE ISW-BC

Closest papers by meaning

• Dual RL: Unification and New Methods for Reinforcement and Imitation Learning (ICLR 2024 · similarity 0.93)
• Hybrid Inverse Reinforcement Learning (ICML 2024 · similarity 0.89)
• Learning on One Mode: Addressing Multi-modality in Offline Reinforcement Learning (ICLR 2025 · similarity 0.88)
• Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective (ICML 2024 · similarity 0.88)
• Is Value Learning Really the Main Bottleneck in Offline RL? (NeurIPS 2024 · similarity 0.87)
• Blending Imitation and Reinforcement Learning for Robust Policy Improvement (ICLR 2024 · similarity 0.87)