Research map / Reinforcement learning / Offline RL
Conservative Offline Goal-Conditioned Implicit V-Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CGCIVL, an offline goal-conditioned RL method that penalizes values of unconnected state-goal pairs and uses a quasimetric network distilled from a TD-learned value network to handle cross-trajectory goal stitching, evaluated on OGBench.
Problem
Offline goal-conditioned RL fails on goal-stitching tasks requiring cross-trajectory state-goal pairs; naive cross-trajectory sampling causes value overestimation on unconnected pairs that propagates through TD bootstrapping.
Approach
Adds a penalty term that pushes down values of randomly sampled (unconnected) state-goal pairs within an implicit V-learning framework, and uses a quasimetric value network (IQE) distilled from an unconstrained TD network so that connected cross-trajectory values are bounded via the triangle inequality.
Key results
- On PointMaze-giant-stitch, CGCIVL reaches 81% success versus 50% for the second-best method (QRL).
- Across OGBench locomotion tasks, CGCIVL averages 89.33% on PointMaze stitch tasks versus 71.33% for QRL and 86.33% on PointMaze navigate tasks.
- Ablations show removing quasimetric distillation severely hurts stitching performance, and η=0 (no penalty) degrades both navigation and stitching.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CGCIVL instead of QRL | Offline goal-conditioned learning · pointmaze-giant-stitch-v0 | success rate (%): 50 ± 8 -> 81 ± 17 | Helped |
| Use CGCIVL instead of HIQL | Offline goal-conditioned learning · antmaze-giant-stitch-v0 | success rate (%): 25 ± 7 -> 36 ± 7 | Helped |
| Use CGCIVL instead of HIQL | Offline goal-conditioned learning · humanoidmaze-giant-stitch-v0 | success rate (%): 6 ± 5 -> 34 ± 6 | Helped |
| Use CGCIVL instead of GCIQL | Offline goal-conditioned learning · cube-single-play-v0 | success rate (%): 68 ± 6 -> 84 ± 4 | Helped |
| Use CGCIVL instead of GCIQL | Offline goal-conditioned learning · scene-play-v0 | success rate (%): 51 ± 4 -> 77 ± 5 | Helped |
| Lower the in-trajectory goal-sampling weight α (ablation) | CGCIVL · goal-stitching tasks | success rate | Helped |
| Raise the in-trajectory goal-sampling weight α (ablation) | CGCIVL · goal-navigating tasks | success rate | Helped |
| Remove the conservative penalty (η = 0.0) (ablation) | CGCIVL · pointmaze-giant-navigate and pointmaze-giant-stitch | success rate | Hurt |
| Use an excessively large penalty coefficient (η = 1.0) (ablation) | CGCIVL · pointmaze-giant-navigate and pointmaze-giant-stitch | success rate | Hurt |
| Remove the hierarchical policy framework (ablation) | CGCIVL · long-horizon PointMaze tasks | success rate | Hurt |
| Remove quasimetric distillation (ablation) | CGCIVL · PointMaze goal-stitching tasks | success rate | Hurt |
Surprising findings
- Directly sampling cross-trajectory goals (lower α) degrades performance even on goal-navigating tasks, as the paper shows overestimation of unconnected pairs.
Limitations
- Relies on a hand-chosen penalty coefficient η and goal-sampling ratio α, selected by empirical validation over candidate values.
- Does not reach the best results on several Cube, Puzzle and AntMaze-giant stitch tasks, where HIQL or GCIQL is competitive.
Experimental setup
Datasets: OGBench PointMaze OGBench AntMaze OGBench HumanoidMaze OGBench Cube OGBench Scene OGBench Puzzle
Benchmarks: OGBench
Metrics: average binary success rate
Baselines: GCBC GCIVL GCIQL QRL CRL HIQL
Builds on: Goal-Conditioned Implicit V-Learning Implicit Q-Learning Interval Quasimetric Embeddings Hindsight Experience Replay Quasimetric Reinforcement Learning
Closest papers by meaning
- Stitching Sub-trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL (AAAI 2024 · similarity 0.90)
- Physics-informed Value Learner for Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
- OGBench: Benchmarking Offline Goal-Conditioned RL (ICLR 2025 · similarity 0.89)
- Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Goal-Conditioned On-Policy Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Flattening Hierarchies with Policy Bootstrapping (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.