Research map / Reinforcement learning / Offline RL
Adaptive $Q$-Aid for Conditional Supervised Learning in Offline Reinforcement Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes QCS, which adds an IQL-pretrained Q-function as a return-weighted auxiliary term to the RCSL (Decision ConvFormer/DT/MLP) loss, improving stitching and beating RCSL and value-based baselines on D4RL MuJoCo, AntMaze, and Adroit.
Problem
Return-conditioned supervised learning (RCSL) in offline RL lacks stitching ability, so it is limited to the best trajectories in a dataset, while Q-functions learned from near-optimal data over-generalize and can hurt stable learning.
Approach
Instead of using the Q-function as a conditioning input, QCS adds a Q-aid term to the RCSL loss, weighted by a linear decreasing function of trajectory return (lambda*(R*-R(tau))), so that optimal trajectories follow pure RCSL and suboptimal ones draw more on the IQL Q-function.
Key results
- QCS-R reaches 90.3 average normalized score on MuJoCo versus 83.1 for the best baseline (SQL) and 82.4 for ACT, per Table 2.
- On AntMaze, QCS-G averages 80.4 versus 76.2 for POR-style combined methods and 71.8 for SQL; on antmaze-large-diverse it scores 77.3 vs. 73.4 for DC-style baselines.
- Pen-human and pen-cloned reach 83.9 and 66.5 with QCS-R, beating DC (74.2, 50.0).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add dynamically trajectory-return-weighted Q-aid to RCSL (ablation) | QCS-R · MuJoCo medium | normalized return: 74.7 -> 81.2 | Helped |
| add dynamically trajectory-return-weighted Q-aid to RCSL (ablation) | QCS-R · MuJoCo medium-replay | normalized return: 75.4 -> 82.9 | Helped |
| add dynamically trajectory-return-weighted Q-aid to RCSL (ablation) | QCS-R · MuJoCo medium-expert | normalized return: 104.2 -> 106.7 | Helped |
| increase λ from 0.5 to 1.5 (ablation) | QCS-R · Walker2d medium | normalized return: 88.2 ± 1.1 -> 60.7 ± 11.2 | Hurt |
| use maximum dataset trajectory return instead of optimal environment return for R* (ablation) | QCS-R · HalfCheetah medium | normalized return: 59.0 ± 0.4 -> 55.2 ± 0.5 | Hurt |
| replace IQL-trained Q-function with CQL-trained Q-function (ablation) | QCS-R · MuJoCo medium | normalized return: 58.3 ± 1.2 -> 68.1 ± 1.5 | Helped |
| replace IQL-trained Q-function with CQL-trained Q-function (ablation) | QCS-R · AntMaze umaze-diverse | normalized return: 84.0 -> 85.2 | Helped |
| add goal conditioning (ablation) | QCS DC-based · AntMaze large-play | normalized return: 41.2 -> 70.0 | Helped |
| use DC instead of DT as the base architecture (ablation) | QCS · HalfCheetah medium | normalized return: 58.7 -> 59.0 | No clear effect |
| combine RCSL with Q-aid | QCS-R · HalfCheetah medium | normalized return: 42.6 -> 59.0 ± 0.4 | Helped |
| combine RCSL with Q-aid | QCS-G · AntMaze large-play | normalized return: 58.0 -> 70.0 ± 9.6 | Helped |
Surprising findings
- A constant Q-aid weight is inferior to the return-dependent weight, and a Q-function trained with CQL gives weaker gains than with IQL.
- Plain DT/DC and max-Q trade off opposite ways across dataset qualities: max-Q performs poorly on expert-quality data (e.g., hopper-expert near 1.8), while RCSL excels there.
Limitations
- The linear weight function and the choice of R* (optimal environment return or maximum dataset return) are heuristic and sensitive to lambda, e.g., walker2d-medium degrades for lambda above 0.5.
- Q-aid relies on a separately pretrained IQL Q-function, and gains from other Q-learning methods (CQL) are smaller.
Experimental setup
Datasets: D4RL MuJoCo v2 medium D4RL MuJoCo v2 medium-replay D4RL MuJoCo v2 medium-expert D4RL AntMaze v2 D4RL Adroit pen human and cloned Inverted Double Pendulum datasets generated with SAC
Benchmarks: D4RL MuJoCo D4RL AntMaze D4RL Adroit
Metrics: expert-normalized return mean and std over 5 seeds
Baselines: TD3+BC IQL CQL SQL DT DC RvS QDT EDT CGDT ACT POR
Builds on: Decision Transformer Decision ConvFormer Implicit Q-Learning RvS
Closest papers by meaning
- Q-Supervised Contrastive Representation: A State Decoupling Framework for Safe Offline Reinforcement Learning (ICML 2025 · similarity 0.89)
- Exclusively Penalized Q-learning for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- On the Complexity of Offline Reinforcement Learning with Q*-Approximation and Partial Coverage (NeurIPS 2026 · similarity 0.88)
- Q-Distribution guided Q-learning for offline reinforcement learning: Uncertainty penalized Q-value via consistency model (NeurIPS 2024 · similarity 0.88)
- C2IQL: Constraint-Conditioned Implicit Q-learning for Safe Offline Reinforcement Learning (ICML 2025 · similarity 0.88)
- Flow Q-Learning (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.