Research map / Reinforcement learning / Visual and representation RL
Do's and Don'ts: Learning Desirable Skills with Instruction Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DoDont, which trains an instruction network on a few action-free Do's/Don'ts videos and uses it as the distance function (multiplicatively weighting the intrinsic reward) in distance-maximizing skill discovery (METRA/HILP) to learn desirable skills and avoid unsafe ones.
Problem
Unsupervised skill discovery struggles to learn complex locomotion (walking, running) in high-dimensional continuous control and often learns unsafe or undesirable behaviors such as flipping, rolling, or entering hazardous regions, while hand-designed reward functions are hard to scale to multiple b
Approach
Learn a reward-like instruction network from action-free Do's and Don'ts video transitions via binary classification, then freeze it and use it to multiplicatively weight the METRA intrinsic reward (reformulated constraint so the signal acts directly), rather than adding it to the reward as in SMERL/DGPO.
Key results
- In online DMC locomotion with only 8 instruction videos, DoDont achieves higher running reward and state coverage than METRA, with METRA's Quadruped skills primarily rolling.
- In offline zero-shot RL aggregated over 128 values, DoDont substantially outperforms HILP, FB, and FDM (e.g., APS Walker Walk 930 vs HILP 862 and Cheetah Walk 984 vs 939).
- In the Kitchen manipulation environment, DoDont completes more of the six tasks than METRA, while SMERL and DGPO complete only two to three.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use video-trained instruction network to weight the METRA skill-discovery reward | DoDont · Cheetah and Quadruped running tasks | zero-shot running reward | Helped |
| Use video-trained instruction network to guide skill discovery | DoDont · Cheetah and Quadruped locomotion | state coverage | Helped |
| Use video-trained instruction network to guide skill discovery | DoDont · safe-area navigation | safe state coverage | Helped |
| Use Do and Don't videos to guide locomotion skills | DoDont · Quadruped and Cheetah running while avoiding rolling or flipping | zero-shot reward | Helped |
| Use video-trained instruction network to guide manipulation skills | DoDont · Kitchen | policy task coverage | Helped |
| Use the instruction network with offline skill learning | DoDont · offline zero-shot RL | interquartile mean | Helped |
| Use only one Do video (and one Don't video) instead of the default video set (ablation) | DoDont · Quadruped running | zero-shot run reward | No clear effect |
| Replace the direct instruction-weighted reward with the delayed reward formulation (ablation) | DoDont · Quadruped running | zero-shot reward | Hurt |
| Remove penalties for Don't behaviors and encourage only Do behaviors (ablation) | DoDont · Quadruped running | zero-shot Run return | Hurt |
| Add the instruction-network score to the intrinsic METRA reward instead of multiplying it (ablation) | DoDont · Cheetah safe-area navigation | qualitative skill performance | Hurt |
| Use hand-designed run reward as the distance metric | METRA dagger · Cheetah running | running behavior learning | Helped |
| Use hand-designed run reward as the distance metric | METRA dagger · Quadruped running | running behavior learning | Hurt |
Surprising findings
- Using only one Do's video, the agent still learned movement in multiple directions in the Quadruped environment.
- Multiplicative weighting of the intrinsic reward outperformed additive combination with the instruction network, which the authors attribute to scale control.
Limitations
- Instruction network is trained on in-domain video data, which is not readily available in real-world settings.
- Experiments are in simulation (DMC, Kitchen) with pixel and state inputs rather than real robots.
- Some offline zero-shot comparisons are weaker than HILP on certain tasks (e.g., Quadruped Walk 568 vs 605 for HILP on APS).
Experimental setup
Datasets: D4RL ExORL DMC Cheetah/Quadruped expert SAC rollouts Kitchen random action rollouts
Benchmarks: DMC Cheetah DMC Quadruped Kitchen Walker Jaco ExORL
Metrics: state coverage zero-shot task reward safe state coverage policy task coverage interquartile mean (IQM) zero-shot return downstream task reward
Baselines: METRA METRA† SMERL DGPO FDM FB HILP
Builds on: METRA HILP SMERL DGPO Wasserstein dependency measure
Closest papers by meaning
- Learning to Discover Skills through Guidance (NeurIPS 2023 · similarity 0.89)
- VICtoR: Learning Hierarchical Vision-Instruction Correlation Rewards for Long-horizon Manipulation (ICLR 2025 · similarity 0.86)
- Periodic Skill Discovery (NeurIPS 2025 · similarity 0.85)
- Skill Disentanglement in Reproducing Kernel Hilbert Space (AAAI 2025 · similarity 0.85)
- DIDI: Diffusion-Guided Diversity for Offline Behavioral Generation (ICML 2024 · similarity 0.85)
- Efficient Skill Discovery via Regret-Aware Optimization (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.