Research map / Reinforcement learning / Offline RL
ContraDiff: Planning Towards High Return States via Contrastive Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CDiffuser, a diffusion-based offline RL planner that adds a contrastive loss pulling generated states toward high-return dataset states and away from low-return ones, improving performance when high-return trajectories are scarce.
Problem
Offline RL performance degrades when the dataset contains mostly low-return trajectories, so few high-return examples are available to learn from.
Approach
Uses contrastive learning over state returns (not trajectory labels) to constrain diffusion-generated subsequent trajectories toward high-return states and away from low-return states, built on Diffuser with a return predictor for guidance; two sampling strategies (SR and SRD) pick contrastive samples.
Key results
- CDiffuser-SR reaches 142.9 on Maze2d U-Maze vs 113.9 for Diffuser, and 92.3 on Hopper-Medium vs 74.3 for Diffuser.
- On HalfCheetah Random mixed with 0.3 expert data, CDiffuser-SRD scores 88.7 vs 75.8 for Diffuser (about 12.9 improvement).
- Transplanting the contrastive module improves Diffuser on all three Med-Expert tasks (e.g., +9.1 on Hopper).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add a contrastive mechanism to Diffuser | Diffuser on D4RL benchmark tasks · D4RL benchmarks | normalized score | Mixed |
| add the contrastive mechanism (ablation) | CDiffuser · nine ablation tasks | normalized score | Helped |
| use low-return trajectories rather than training solely on high-return samples (ablation) | CDiffuser · nine ablation tasks | normalized score | Helped |
| use SR rather than SRD sampling (ablation) | CDiffuser · D4RL benchmark tasks | normalized score | Mixed |
| transplant the contrastive mechanism | Decision Diffuser · HalfCheetah Med-Expert | normalized score improvement: 0 | No clear effect |
Surprising findings
- CDiffuser-SRD underperforms CDiffuser-SR on several Med-Expert and Med-Replay settings, so the best sampling strategy is task-dependent.
- The contrastive transplant to Decision Diffuser yields no improvement on HalfCheetah Med-Expert.
Limitations
- Contrastive mechanism only applied over state returns; action or latent-level contrast left unexplored.
- Gains are inconsistent across datasets; SRD underperforms SR in several settings and the choice between them is made by visual inspection.
- Reports standard deviations inconsistently (10 vs 50 seeds stated in different tables).
Experimental setup
Datasets: D4RL HalfCheetah D4RL Hopper D4RL Walker2d D4RL Maze2d D4RL Kitchen mixed HalfCheetah datasets with expert ratios
Benchmarks: D4RL HalfCheetah D4RL Hopper D4RL Walker2d D4RL Maze2d D4RL Kitchen
Metrics: normalized average return
Baselines: CQL IQL MOPO Decision Transformer Trajectory Transformer Diffuser Decision Diffuser
Builds on: Diffuser Decision Diffuser infoNCE
Closest papers by meaning
- DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitching (ICML 2024 · similarity 0.89)
- Exploiting Action Impact Regularity and Exogenous State Variables for Offline Reinforcement Learning (Abstract Reprint) (AAAI 2024 · similarity 0.89)
- ODRL: A Benchmark for Off-Dynamics Reinforcement Learning (NeurIPS 2024 · similarity 0.89)
- Forecasting in Offline Reinforcement Learning for Non-stationary Environments (NeurIPS 2025 · similarity 0.88)
- A Clean Slate for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Compositional Conservatism: A Transductive Approach in Offline Reinforcement Learning (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.