Research map / Time series / Causal and generative time series
Contrastive Difference Predictive Coding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces TD InfoNCE, a temporal-difference version of contrastive predictive coding that estimates discounted state occupancy measures off-policy, and applies it to goal-conditioned RL.
Problem
Learning representations that encode long-horizon temporal dependencies with contrastive predictive coding requires large amounts of data; goal-conditioned RL can be cast as predicting future state visitation.
Approach
Derives a temporal-difference estimator of the InfoNCE loss by expanding the Monte Carlo InfoNCE via the discounted state occupancy recursion and replacing on-policy future-state sampling with a softmax-based importance weight, yielding an off-policy, nonparametric successor-representation-like method used as an actor-critic for goal-conditioned RL.
Key results
- On challenging Fetch tasks (pick & place, slide), TD InfoNCE achieves about 2x median improvement in success rate over the strongest baseline (QRL).
- In the tabular gridworld, TD InfoNCE is about 1500x more sample efficient than Monte Carlo InfoNCE (6.5e3 vs 1e7 transitions) and about 20x more than the successor representation.
- On D4RL AntMaze, TD InfoNCE matches or outperforms baselines on 5 of 6 tasks (e.g., 91.7 on umaze-diverse vs 80.1 for QRL).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Monte Carlo InfoNCE with temporal-difference InfoNCE | Contrastive RL · AntMaze medium-play-v2 and medium-diverse-v2 | success rate: +13% | Helped |
| Use TD InfoNCE instead of Monte Carlo InfoNCE | Contrastive RL · Tabular gridworld discounted state occupancy estimation | sample efficiency: $1500\times$ more sample efficient ($6.5 \times 10^3$ vs $10^7$ transitions) | Helped |
| Use TD InfoNCE instead of C-learning | C-learning · Tabular gridworld discounted state occupancy estimation | sample efficiency: $130\times$ less data ($7.7 \times 10^4$ vs $10^7$ transitions) | Helped |
| Use TD InfoNCE instead of successor representations | Successor representations · Tabular gridworld discounted state occupancy estimation | sample efficiency: almost $20 \times$ fewer samples ($5.2 \times 10^5$ vs $10^7$ transitions) | Helped |
| Use categorical cross-entropy instead of binary cross-entropy (ablation) | TD InfoNCE versus C-learning · Tabular discounted state occupancy estimation | estimation error | Helped |
| Use self-normalized softmax importance weights instead of unnormalized importance weights (ablation) | TD InfoNCE · Tabular discounted state occupancy estimation | performance | No clear effect |
| Use N² negative examples instead of N negative examples (ablation) | TD InfoNCE · Tabular discounted state occupancy estimation | performance | No clear effect |
| Change actor learning rate from $5 \times 10^{-5}$ to $1 \times 10^{-4}$ (ablation) | TD InfoNCE · Push (state) and slide (state) | success rate | No clear effect |
Surprising findings
- Ablations suggest the categorical (softmax) cross-entropy loss, rather than N^2 negatives or self-normalized weights, accounts for most of the gain over C-learning.
Limitations
- More complex and more hyperparameters than the Monte Carlo counterpart; computes loss over N^2 negatives.
- Still struggles on the most challenging image-based control tasks.
- Convergence proof uses a modified loss (sum replaced by expectation).
Experimental setup
Datasets: Fetch robotics benchmark (online GCRL, state and image) D4RL AntMaze 5x5 gridworld (uniform random policy)
Benchmarks: Fetch robotics GCRL (push, pick & place, slide; state and image) D4RL AntMaze (umaze, medium, large) 5x5 gridworld state occupancy estimation
Metrics: success rate absolute error of discounted state occupancy minimum distance to goal
Baselines: QRL contrastive RL contrastive RL (NCE) GCBC DDPG + HER DT IQL TD3 + BC C-learning successor representation
Builds on: InfoNCE / contrastive predictive coding contrastive RL successor representation C-learning quasimetric RL
Closest papers by meaning
- Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inference (NeurIPS 2024 · similarity 0.88)
- Self-Supervised Contrastive Learning for Long-term Forecasting (ICLR 2024 · similarity 0.86)
- Towards Principled Representation Learning from Videos for Reinforcement Learning (ICLR 2024 · similarity 0.85)
- Offline Goal-conditioned Reinforcement Learning with Quasimetric Representations (NeurIPS 2025 · similarity 0.85)
- Predictive Coding Enhances Meta-RL To Achieve Interpretable Bayes-Optimal Belief Representation Under Partial Observability (NeurIPS 2025 · similarity 0.85)
- Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.