Research map / Reinforcement learning / Imitation learning
Enhancing Online Reinforcement Learning with Meta-Learned Objective from Offline Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GILD, a meta-learned general imitation-learning objective from sub-optimal offline demonstrations that augments off-policy RL (DDPG, TD3, SAC) under sparse MuJoCo rewards via bi-level optimization.
Problem
Online RL with sparse rewards struggles; conventional RL+IL uses a handcrafted imitation objective that restricts the policy to be sub-optimal when offline demonstrations are generated by non-expert policies, and balancing RL and IL requires heavy hyperparameter tuning.
Approach
Replace the handcrafted behavior-cloning term with a neural objective L_GILD(phi) = f_omega(s_d, a_d, phi(s_d)) meta-learned at the upper level by a meta-loss that compares Q-values of the RL+GILD actor against a pseudo-updated RL+IL actor, so the GILD objective distills demonstration knowledge without pinning the policy to demonstrator behavior. GILD is a small three-layer network with no domain-
Key results
- TD3+GILD reaches 3538.6 on sparse Hopper-v2 and SAC+GILD reaches 11161.5 on sparse HalfCheetah-v2 (vs. 5843.9 for TD3+IL and 6503.2 for SAC+IL on HalfCheetah), with GILD variants best in all four MuJoCo tasks.
- On sparse Ant-v2, SAC+GILD achieves 5335.3 vs. 3370.8 for SAC+IL and 4589.5 for LOGO.
- GILD converges within about 1% of total training steps; RL+GILD with 1% warm-start cuts run time to roughly the level of vanilla RL (e.g., SAC+GILD+1%ws 4h59m vs. SAC 4h40m average).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace conventional IL with meta-learned GILD | DDPG · MuJoCo sparse-reward tasks and Point2D Navigation | max average return | Helped |
| replace conventional IL with meta-learned GILD | TD3 · MuJoCo sparse-reward tasks and Point2D Navigation | max average return | Helped |
| replace conventional IL with meta-learned GILD | SAC · MuJoCo sparse-reward tasks and Point2D Navigation | max average return | Helped |
| use the meta-loss in Eq. [ref] rather than the intuitive meta-loss (ablation) | DDPG+GILD · Ant-v2 | max average return: 883.1+-254.8 -> 971.6+-296.7 | Helped |
| use the meta-loss in Eq. [ref] rather than the intuitive meta-loss (ablation) | TD3+GILD · Ant-v2 | max average return: 4259.4+-716.3 -> 4864.6+-699.1 | Helped |
| use the meta-loss in Eq. [ref] rather than the intuitive meta-loss (ablation) | SAC+GILD · Ant-v2 | max average return: 4851.0+-218.5 -> 5335.3+-246.9 | Helped |
| add a 1% GILD warm-start, then continue with vanilla RL (ablation) | DDPG+GILD · MuJoCo tasks | average run time: 4h21m -> 2h5m | Helped |
| add a 1% GILD warm-start, then continue with vanilla RL (ablation) | TD3+GILD · MuJoCo tasks | average run time: 4h35m -> 2h18m | Helped |
| add a 1% GILD warm-start, then continue with vanilla RL (ablation) | SAC+GILD · MuJoCo tasks | average run time: 9h24m -> 4h59m | Helped |
| increase GILD warm-start steps beyond 1% (ablation) | RL+GILD · Ant-v2 | max average return | Helped |
Surprising findings
- Meta-learned objective with a meta-loss outperformed an intuitive meta-loss that directly maximizes Q of the GILD-updated policy, and the GILD objective improved vanilla RL even when only used for 1% warm-start steps.
Limitations
- Single-task meta-RL framework: GILD must learn from scratch for each unseen task, limiting few-shot use.
- Evaluated only on four MuJoCo sparse-reward tasks and a 2D navigation toy; complex tasks such as image-based control may converge slower due to larger offline data.
- Meta-loss and second-order gradient computation add cost compared with RL+IL (about 1.5-2x run time without warm-start).
Experimental setup
Datasets: Hopper-v2 Walker2d-v2 HalfCheetah-v2 Ant-v2 Point2D Navigation
Benchmarks: Hopper-v2 sparse Walker2d-v2 sparse HalfCheetah-v2 sparse Ant-v2 sparse Point2D Navigation
Metrics: max average return average normalized score (Expert=100) run time KL divergence to Behavior policy
Baselines: DDPG DDPG+IL TD3 TD3+IL SAC SAC+IL PPO PPO+IL LOGO Meta-Critic DiffAIL
Builds on: TD3+BC LOGO Meta-Critic EMRLD DDPG TD3 SAC
Closest papers by meaning
- Hybrid Inverse Reinforcement Learning (ICML 2024 · similarity 0.91)
- SEABO: A Simple Search-Based Method for Offline Imitation Learning (ICLR 2024 · similarity 0.90)
- Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective (ICML 2024 · similarity 0.90)
- RLIF: Interactive Imitation Learning as Reinforcement Learning (ICLR 2024 · similarity 0.90)
- Blending Imitation and Reinforcement Learning for Robust Policy Improvement (ICLR 2024 · similarity 0.90)
- Dual RL: Unification and New Methods for Reinforcement and Imitation Learning (ICLR 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.