Research map / Probabilistic and scientific modeling / Molecules and proteins
Temporal Difference Flows
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Temporal Difference Flows (TD-Flow), flow-matching and diffusion methods that learn geometric horizon models (successor measure) using the Bellman structure in both sampling and objective, giving stable long-horizon predictions.
Problem
Learning accurate generative models of the successor measure (geometric horizon models) for long horizons is unstable because bootstrapped TD training compounds errors and grows inaccurate beyond roughly 20-50 steps.
Approach
Exploit the iterative nature of flow matching and diffusion by making the Bellman operator act on probability paths: the bootstrapped term uses the previous iteration's learned flow/ODE both to sample intermediate states and as the regression target (TD^2-CFM), with a proof that the iteration converges to the successor measure and a variance analysis showing lower gradient variance than naive TD-C
Key results
- TD2-CFM (TD-Flow) achieves about 10x reduction in value-function MSE, 1.5x lower EMD, and 3x lower log-likelihood than TD-CFM at effective horizon 100, averaged across four domains.
- At effective horizon 100 on the Pointmass loop task, TD2 methods keep accuracy while naive methods degrade by nearly four orders of magnitude in value MSE.
- TD-based GHM planning via GPI gives roughly 30%+ improvement over zero-shot Forward-Backward policies for TD-CFM(c) and TD2-CFM across all sampling strategies.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TD²-CFM instead of TD-CFM | TD-CFM · 19 tasks, 4 domains | value-function MSE: 10× reduction | Helped |
| Use TD²-CFM instead of TD-CFM | TD-CFM · 19 tasks, 4 domains | EMD: 1.5× reduction | Helped |
| Use TD²-CFM instead of TD-CFM | TD-CFM · 19 tasks, 4 domains | log-likelihood: 3× reduction | Helped |
| Use TD²-based methods rather than naive implementations | GHM methods · Pointmass loop, effective horizon 100 | value-function MSE: nearly four orders of magnitude | Helped |
| Use a straight conditional path in TD-CFM(c) (ablation) | TD-CFM(c) · 19 tasks, 4 domains | value-function MSE | Helped |
| Replace the straight conditional path with a curved path in TD²-CFM (ablation) | TD²-CFM · 19 tasks, 4 domains | performance | Helped |
| Replace the straight conditional path with a curved path in TD-CFM(c) (ablation) | TD-CFM(c) · 19 tasks, 4 domains | performance | Hurt |
| Use flow matching rather than diffusion | TD-based generative models · 19 tasks, 4 domains | performance across metrics | Helped |
| Use TD²-CFM for generalized policy improvement instead of the FB policy | FB policy · 22 tasks, 4 domains | return improvement: ≈30%+ | Helped |
| Use FB-GPI instead of the zero-shot FB policy | FB policy · 22 tasks, 4 domains | return | Hurt |
| Increase the effective horizon | TD²-based methods · Pointmass loop | value-function MSE | No clear effect |
| Increase the effective horizon | alternative methods · Pointmass loop | value-function MSE | Hurt |
Surprising findings
- With curved conditional paths, TD2-CFM maintained or improved performance while TD-CFM(c) degraded significantly.
- FB-based GPI fails to improve and on average deteriorates performance, with degradation in three of four domains.
Limitations
- Iterative sampling via neural ODE/SDE is computationally costly at inference and training.
- Theoretical guarantees hold for idealized exact minimization of the loss at each iteration.
- Experiments limited to DeepMind Control Suite state-based tasks with a single or FB-conditioned policy set.
Experimental setup
Datasets: ExoRL DeepMind Control Suite
Benchmarks: DeepMind Control Suite (Maze, Walker, Cheetah, Quadruped; 22 tasks)
Metrics: normalized negative log-likelihood earth mover's distance value function MSE GPI return improvement
Baselines: TD-CFM TD-CFM(c) TD-DD GAN GHM beta-VAE GHM FB-GPI
Builds on: flow matching conditional flow matching denoising diffusion successor measure generalized policy improvement forward-backward representation
Closest papers by meaning
- Discerning Temporal Difference Learning (AAAI 2024 · similarity 0.88)
- Beyond Prediction: Steering VLM Agents with Retrospective World Modeling (NeurIPS 2026 · similarity 0.87)
- Latent Plan Transformer for Trajectory Abstraction: Planning as Latent Space Inference (NeurIPS 2024 · similarity 0.87)
- Composite Flow Matching for Reinforcement Learning with Shifted-Dynamics Data (NeurIPS 2025 · similarity 0.86)
- Predictive but Not Plannable: RC-aux for Latent World Models (NeurIPS 2026 · similarity 0.86)
- Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Making (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.