Research map / Reinforcement learning / Offline RL

How to Solve Contextual Goal-Oriented Problems with Offline Datasets?

NeurIPS 2024 · Reinforcement learning › Offline RL · approach: Diffusion models for offline decision making · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CODA, which converts unlabeled dynamics trajectories and positive context-goal pairs into a fully labeled transition dataset of an action-augmented MDP so standard Bellman-based offline RL (IQL, PSPI) can solve contextual goal-oriented problems without negative data.

Problem

Solving contextual goal-oriented (CGO) problems offline when only unlabeled dynamics trajectories and positive context-goal pairs are available, with no negative (non-goal) examples and no reward labels for trajectories.

Approach

Augment the action space with a fictitious action a+ that is rewarded only in goal sets and transitions to an absorbing terminal state; this makes the augmented MDP equivalent to the original, so context-goal pairs yield fictitious rewarded transitions and dynamics data get labeled with zero reward under sampled contexts, producing a fully labeled dataset with no extra approximation error.

Key results

• On original AntMaze, CODA averages 70.8% success vs 50.5% for PDS, 31.2% for goal prediction, and 66.6% for oracle reward.
• On Four Rooms with perturbation, CODA averages 75.0% vs 30.6% for PDS and 52.4% for goal prediction.
• On Random Cells, CODA averages 66.8% vs 55.5% for PDS and 55.8% for goal prediction; under context distribution shift it reaches 64.7% vs 49.9% for PDS.

Measured effects reported in the paper

Surprising findings

• Training with the oracle reward does not generally improve over CODA, which the authors attribute to sparsity of positive samples in randomly paired context-transition data.
• UDS and naive reward prediction collapse to near zero on several medium and large mazes, so they were omitted from later experiments.

Limitations

• Experiments are limited to low-dimensional AntMaze simulations.
• Theory is instantiated with PSPI; analysis for other base algorithms such as IQL is not performed.
• Requires concentrability coefficients to be finite, i.e. coverage of the optimal policy's trajectories and goals by the datasets.

Experimental setup

Datasets: D4RL AntMaze-v2 (dynamics data, rewards and terminals removed) constructed context-goal datasets (Original AntMaze, Four Rooms, Random Cells)

Benchmarks: AntMaze-v2 (original) Four Rooms with perturbation Random Cells Random Cells with perturbation

Metrics: average success rate (%) normalized return

Baselines: PDS Goal Prediction RP UDS+RP Oracle Reward

Builds on: IQL PSPI MAHALO PDS HER

Closest papers by meaning

• MGDA: Model-based Goal Data Augmentation for Offline Goal-conditioned Weighted Supervised Learning (AAAI 2025 · similarity 0.85)
• Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learning (ICML 2025 · similarity 0.83)
• GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learning (NeurIPS 2024 · similarity 0.82)
• Causal Action Influence Aware Counterfactual Data Augmentation (ICML 2024 · similarity 0.81)
• Stitching Sub-trajectories with Conditional Diffusion Model for Goal-Conditioned Offline RL (AAAI 2024 · similarity 0.81)
• Few-Shot Knowledge Distillation of LLMs With Counterfactual Explanations (NeurIPS 2025 · similarity 0.81)