Research map / Reinforcement learning / Offline RL
Model-based Offline Reinforcement Learning with Lower Expectile Q-Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Lower Expectile Q-learning (LEQ), a model-based offline RL method that uses lower-expectile regression of lambda-returns over imagined rollouts to get conservative value estimates, achieving strong AntMaze results.
Problem
Model-based offline RL suffers from inaccurate, overestimated value estimates from learned-model rollouts, especially in long-horizon sparse-reward tasks where uncertainty-based penalties give sub-optimal behavior.
Approach
Replace uncertainty-based Q-penalties with lower-expectile (tau<0.5) regression for the critic and policy, applied to lambda-returns of 10-step model rollouts, so that the value estimate is conservative and low-bias without exhaustive distributional computation.
Key results
- LEQ scores 60.2 on antmaze-large-diverse and 55.8 on antmaze-ultra-diverse, versus 4.0 and 3.6 for the next-best model-based method IQL-TD-MPC; total AntMaze score 470.4 (ablation table) vs 348.5 for IQL-TD-MPC.
- On D4RL MuJoCo, LEQ totals 923.2 vs 959.5 for MOBILE and 953.4 for CBOP; on NeoRL it totals 572.4, the highest of the compared methods.
- On V-D4RL with DreamerV3, LEQ totals 307.6, close to the best model-free and model-based scores (ROSMO 310.9, DrQ+BC 321.6).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use λ-returns instead of H-step returns for critic and policy updates (ablation) | LEQ · AntMaze total | success rate total: 290.3 -> 461.8 | Helped |
| use λ-returns instead of 1-step critic updates and direct-Q policy optimization (ablation) | LEQ · AntMaze total | success rate total: 373.5 -> 461.8 | Helped |
| use lower-expectile λ-return policy learning instead of direct Q(s,a) maximization (ablation) | LEQ · AntMaze total | success rate total: 410.3 -> 461.8 | Helped |
| use lower-expectile λ-return policy learning instead of AWR (ablation) | LEQ · AntMaze total | success rate total: 114.2 -> 461.8 | Helped |
| use lower-expectile Q-learning instead of MOBILE conservatism (ablation) | LEQ with λ-returns · AntMaze total | success rate total: 338.9 -> 461.8 | Helped |
| use lower-expectile Q-learning instead of MOBILE conservatism (ablation) | LEQ without λ-returns · AntMaze total | success rate total: 339.0 -> 373.5 | Helped |
| reduce β from 0.95 to 0.25 (ablation) | MOBILE* · AntMaze total | success rate total: 14.0 -> 213.0 | Helped |
| use γ=0.997 instead of γ=0.99 with β=0.25 and R=10 (ablation) | MOBILE* · AntMaze total | success rate total: 125.7 -> 285.3 | Helped |
| reduce dataset expansion length R from 10 to 5 (ablation) | MOBILE* with β=0.25 and γ=0.997 · AntMaze total | success rate total: 285.3 -> 205.7 | Hurt |
| reduce imagination length H from 10 to 5 (ablation) | LEQ · AntMaze total | success rate total: 470.4 -> 404.8 | Hurt |
| increase imagination length H from 10 to 15 (ablation) | LEQ · AntMaze total | success rate total: 470.4 -> 371.0 | Hurt |
| remove dataset expansion (ablation) | LEQ · AntMaze total | success rate total: 470.4 -> 435.4 | No clear effect |
Surprising findings
- Replacing the ground-truth termination function with a learned one drops LEQ's AntMaze total from 461.8 to 232.6.
- Simply reducing the imaginary-to-real loss ratio beta (to 0.25) makes the MOBILE variant work in AntMaze, suggesting real dataset transitions are undervalued in prior model-based work.
Limitations
- Relies on the ground-truth termination function; a learned termination function reduces AntMaze performance substantially.
- High variance across seeds on antmaze-medium and MuJoCo tasks; policy can collapse to zero after peaks due to optimistic model predictions.
- Model ensembles falsely predict passing through close walls in medium mazes, producing impossible plans.
Experimental setup
Datasets: D4RL AntMaze D4RL MuJoCo Gym NeoRL V-D4RL
Benchmarks: D4RL AntMaze D4RL MuJoCo Gym NeoRL V-D4RL
Metrics: normalized return success rate
Baselines: CQL IQL TT TAP COMBO RAMBO MOBILE CBOP IQL-TD-MPC MOPO EDAC TD3+BC
Builds on: IQL MOBILE CBOP DreamerV3 FQE
Closest papers by meaning
- Offline Reinforcement Learning with Implicit Q-Learning (ICLR 2022 · similarity 0.92)
- Exclusively Penalized Q-learning for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.91)
- A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning (AAAI 2024 · similarity 0.91)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- The Edge-of-Reach Problem in Offline Model-Based Reinforcement Learning (NeurIPS 2024 · similarity 0.90)
- On the Complexity of Offline Reinforcement Learning with Q*-Approximation and Partial Coverage (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.