Research map / Reinforcement learning / Policy gradient and actor-critic
STITCH-OPE: Trajectory Stitching with Guided Diffusion for Off-Policy Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STITCH-OPE, a guided conditional diffusion framework that generates sub-trajectories stitched end-to-end from the target policy, using a negative behavior-policy guidance term, for long-horizon off-policy evaluation.
Problem
Off-policy evaluation in high-dimensional, long-horizon continuous control where importance sampling suffers exponential variance and learned autoregressive models compound errors.
Approach
Train a state-conditioned diffusion model on short sub-trajectories from behavior data, then guide denoising with the target policy score minus a behavior policy score (negative behavior guidance) and stitch sub-trajectories end-to-end, giving an exponential variance reduction in window length rather than full horizon.
Key results
- Across 15 domain-metric instances, STITCH-OPE is best in 11 of 15 (e.g., Hopper log RMSE -2.33 vs PGD -1.22 and MB -1.70).
- On diffusion-policy targets, STITCH-OPE wins 6 of 9 instances, with Hopper rank correlation 0.81 vs 0.68 for MBR.
- Horizon sweep on Hopper-medium shows STITCH-OPE best at every T from 128 to 1000, degrading slowly with T.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use STITCH-OPE instead of competing OPE baselines | OPE on standard benchmark environments · Hopper, Walker2d, HalfCheetah, Pendulum, Acrobot | Log RMSE | Mixed |
| Use STITCH-OPE instead of competing OPE baselines | OPE on standard benchmark environments · Hopper, Walker2d, HalfCheetah, Pendulum, Acrobot | Rank Corr. | Mixed |
| Use STITCH-OPE instead of competing OPE baselines | OPE on standard benchmark environments · Hopper, Walker2d, HalfCheetah, Pendulum, Acrobot | Regret@1 | Mixed |
| Use STITCH-OPE for diffusion target policies instead of baselines | OPE with diffusion policies · D4RL | Log RMSE | Mixed |
| Use STITCH-OPE across increasing evaluation horizons | Hopper-medium · Hopper-medium, T=128–1000 | Log RMSE: -2.23 ± 0.00 -> -4.34 ± 0.01 (T=128); -1.36 ± 0.00 -> -3.19 ± 0.00 (T=256); -0.99 ± 0.00 -> -2.65 ± 0.01 (T=512); -0.81 | Helped |
| Add negative behavior-policy guidance to target-policy guidance (ablation) | STITCH-OPE · GaussianWorld distribution shift | value estimation bias | Helped |
| Use conditional sub-trajectory diffusion and stitching instead of full-length trajectory diffusion (ablation) | STITCH-OPE · GaussianWorld trajectory composition and state generalization | trajectory generation quality | Helped |
| Choose 0 < λ < α for guidance coefficients (ablation) | STITCH-OPE · OPE benchmarks | performance | Helped |
| Set window size w=8 (ablation) | STITCH-OPE · OPE benchmarks | performance | Helped |
| Use STITCH-OPE rather than baselines across Hopper-medium-expert and Hopper-expert datasets (ablation) | OPE · Hopper-medium-expert and Hopper-expert | performance | Helped |
| Use estimated reward instead of the true reward function (ablation) | STITCH-OPE · Hopper-medium | aggregate OPE performance | No clear effect |
| Increase evaluation horizon T (ablation) | STITCH-OPE · Hopper-medium | Log RMSE: -4.34 ± 0.01 -> -1.61 ± 0.02 | Hurt |
Surprising findings
- Full guidance with alpha=lambda=1 is reported as poor, and omitting the behavior guidance term (simple target-score guidance) yields over-regularized sampling.
Limitations
- Theory relies on bounded likelihood ratio and TV-based model fit assumptions; if behavior data has zero-support actions, guidance may produce infeasible trajectories.
- Evaluating trajectory fidelity is hard for complex or partially observable states.
- Performance on domain-specific problems outside robotics is untested.
Experimental setup
Datasets: D4RL halfcheetah-medium D4RL hopper-medium D4RL walker2d-medium OpenAI Gym Pendulum OpenAI Gym Acrobot D4RL hopper-medium-expert D4RL hopper-expert
Benchmarks: D4RL OpenAI Gym Pendulum OpenAI Gym Acrobot
Metrics: Log RMSE Spearman rank correlation Regret@1
Baselines: FQE DR IS DRE MB PGD DWM
Builds on: Policy-Guided Diffusion classifier guidance for diffusion denoising diffusion probabilistic models direct method OPE
Closest papers by meaning
- DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitching (ICML 2024 · similarity 0.88)
- State-Covering Trajectory Stitching for Diffusion Planners (NeurIPS 2025 · similarity 0.88)
- Prior-Guided Diffusion Planning for Offline Reinforcement Learning (NeurIPS 2025 · similarity 0.88)
- Generative Trajectory Stitching through Diffusion Composition (NeurIPS 2025 · similarity 0.88)
- OPERA: Automatic Offline Policy Evaluation with Re-weighted Aggregates of Multiple Estimators (NeurIPS 2024 · similarity 0.88)
- Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.