Research map / Reinforcement learning / Imitation learning
Text-Aware Diffusion for Policy Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TADPoLe, which uses a frozen text-conditioned diffusion model (Stable Diffusion 2.1, and AnimateDiff for video) to compute dense zero-shot text-conditioned rewards for training RL policies (TD-MPC) on Humanoid, Dog, and Meta-World.
Problem
Training RL agents for novel goals or behaviors specified by natural language requires hand-designed reward functions, which do not scale; prior text-to-reward methods rely on CLIP-style models or in-domain demonstrations.
Approach
Treat the policy's rendered frames as samples to be scored by a frozen text-to-image/video diffusion model: the reward is the gap between conditional and unconditional noise predictions (alignment) plus a comparative reconstruction term (naturalness), combined with symlog normalization.
Key results
- On Humanoid standing, TADPoLe reaches 254.43 ground-truth reward vs 247.05 for VLM-RM and 287.68 for the ground-truth reward agent.
- On Humanoid walking, Video-TADPoLe reaches 145.60 vs 25.51 for ViCLIP-RM; on Dog walking it reaches 60.20 vs 14.67 for ViCLIP-RM.
- On 12 Meta-World tasks (average over the 10 tasks reported), TADPoLe averages 50.6% success vs 25.0% for VLM-RM (10-task table, sparse-only baseline 50.6% in appendix).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use TADPoLe instead of VLM-RM for the Humanoid standing reward | TD-MPC · Humanoid standing | cumulative ground-truth reward: 247.05 ($\pm$ 16.90) -> 254.43 ($\pm$ 8.76) | Helped |
| use Video-TADPoLe instead of ViCLIP-RM | TD-MPC · Humanoid walking | cumulative ground-truth reward: 25.51 ($\pm$ 40.45) -> 145.60 ($\pm$ 48.20) | Helped |
| use Video-TADPoLe instead of ViCLIP-RM | TD-MPC · Dog walking | cumulative ground-truth reward: 14.67 ($\pm$ 9.84) -> 60.20 ($\pm$ 8.82) | Helped |
| use TADPoLe instead of VLM-RM | TD-MPC · Meta-World manipulation | average success rate (%): 25.0 -> 50.6 | Helped |
| use symlog normalization instead of direct scaling (ablation) | TADPoLe · Humanoid standing | cumulative ground-truth reward: 239.74 -> 267.23 | Helped |
| use symlog normalization instead of direct scaling (ablation) | Video-TADPoLe · Humanoid walking | cumulative ground-truth reward: 4.58 -> 226.29 | Helped |
| use symlog normalization instead of direct scaling (ablation) | Video-TADPoLe · Dog walking | cumulative ground-truth reward: 35.30 -> 81.22 | Helped |
| replace symlog normalization with SymExp (ablation) | TADPoLe · Humanoid standing | cumulative ground-truth reward: 241.49 -> 267.23 | Helped |
| replace symlog normalization with Min-Max (ablation) | TADPoLe · Humanoid standing | cumulative ground-truth reward: 256.81 -> 267.23 | Helped |
| replace symlog normalization with standardization (ablation) | TADPoLe · Humanoid standing | cumulative ground-truth reward: 236.59 -> 267.23 | Helped |
| set reward weights to w1=2000 and w2=200 instead of w1=200 and w2=2000 (ablation) | TADPoLe · Humanoid stand | cumulative ground-truth reward: 249.82 -> 262.22 | Helped |
| set reward weights to w1=2000 and w2=200 instead of w1=200 and w2=2000 (ablation) | Video-TADPoLe · Humanoid walk | cumulative ground-truth reward: 2.72 -> 226.29 | Helped |
Surprising findings
- Sparse success signal alone matches TADPoLe's average success rate on Meta-World (50.6% vs 42.2% in the appendix 12-task table), so the dense reward is not uniformly better.
- VLM-RM's Meta-World performance drops well below the sparse-only baseline, suggesting CLIP-based dense rewards can hurt policy learning.
Limitations
- Hard to control the weight that individual words in the prompt have on the reward; the agent may remain stationary by focusing on nouns.
- Stochastic resampling of source noise at each timestep causes high variance across runs.
- Human evaluations use 25 participants and small comparisons; ground-truth metrics are unfavorable to vision-only methods.
Experimental setup
Datasets: DeepMind Control Suite Humanoid DeepMind Control Suite Dog Meta-World
Benchmarks: DeepMind Control Suite Humanoid DeepMind Control Suite Dog Meta-World
Metrics: cumulative ground-truth reward success rate human text-alignment preference human naturalness preference
Baselines: VLM-RM LIV Text2Reward ViCLIP-RM Sparse reward
Builds on: DreamFusion TD-MPC Stable Diffusion AnimateDiff symlog
Closest papers by meaning
- RLZero: Direct Policy Inference from Language Without In-Domain Supervision (NeurIPS 2025 · similarity 0.90)
- Text2Reward: Reward Shaping with Language Models for Reinforcement Learning (ICLR 2024 · similarity 0.90)
- Foundation Policies with Hilbert Representations (ICML 2024 · similarity 0.89)
- Conditioning Matters: Training Diffusion Policies is Faster Than You Think (NeurIPS 2025 · similarity 0.89)
- Efficient Language-instructed Skill Acquisition via Reward-Policy Co-Evolution (AAAI 2025 · similarity 0.88)
- Make-An-Agent: A Generalizable Policy Network Generator with Behavior-Prompted Diffusion (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.