Research map / Reinforcement learning / Visual and representation RL
From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following
Justin Fu, Anoop Korattikara, Sergey Levine, Sergio Guadarrama
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Learn a language-conditioned reward and re-plan, rather than directly clone a language-conditioned policy | LC-RL versus Optimal Policy Cloning · Test-House | success rate (%): 8.5 -> 36.4 | Improved |
| Learn a language-conditioned reward with IRL rather than use the GAIL discriminator as reward | LC-RL versus GAIL-Exact · Test-House | success rate (%): 28.3 -> 36.4 | Improved |
| Use oracle ground-truth reward regression instead of IRL reward learning | Reward Regression versus LC-RL · Test-House | success rate (%): 36.4 -> 65.7 | Improved |
| Re-optimize the learned reward with DQN rather than Q-iteration | LC-RL · Test-Task | success rate (%): 51.9 -> 7.7 | Hurt |
| Add value-function reward shaping (ablation) | DQN reward optimization of LC-RL · Test-Task | success rate (%): 7.7 -> 14.1 | Improved |
| Use global pooling rather than a fully connected image embedding (ablation) | LC-RL architecture · Test-Task | success rate (%): 38.4 -> 49.7 | Improved |
| Use global pooling rather than FiLM for the image embedding (ablation) | LC-RL architecture · Test-Task | success rate (%): 43.2 -> 49.7 | Improved |
| Use multiplication instead of addition to combine language and image embeddings (ablation) | Pooling image architecture · Test-Task | success rate (%): 45.5 -> 49.7 | Mixed |
| Use multiplication instead of concatenation to combine language and image embeddings (ablation) | Pooling image architecture · Test-Task | success rate (%): 48.5 -> 49.7 | No clear effect |
| Increase image size from 32 by 24 to 64 by 64 (ablation) | LC-RL with Pooling, Mult · Test-Task | success rate (%): 49.7 -> 51.1 | No clear effect |
| Increase the number of CNN filters or embedding sizes (ablation) | LC-RL · LC-RL environment | performance | No clear effect |
Closest papers by meaning
- GoalLadder: Incremental Goal Discovery with Vision-Language Models (NeurIPS 2025 · similarity 0.91)
- Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning (ICLR 2024 · similarity 0.90)
- ELEMENTAL: Interactive Learning from Demonstrations and Vision-Language Models for Reward Design in Robotics (ICML 2025 · similarity 0.90)
- RLZero: Direct Policy Inference from Language Without In-Domain Supervision (NeurIPS 2025 · similarity 0.90)
- Code as Reward: Empowering Reinforcement Learning with VLMs (ICML 2024 · similarity 0.90)
- Text2Reward: Reward Shaping with Language Models for Reinforcement Learning (ICLR 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
