Research map / Vision-language and multimodal / Robot action and navigation
Guide Your Agent with Adaptive Multimodal Rewards
Changyeon Kim, Younggyo Seo, Hao Liu, Lisa Lee, Jinwoo Shin, Honglak Lee, Kimin Lee
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use adaptive multimodal return-conditioned policy instead of InstructRL | ARP-DT · Procgen CoinRun, Maze I, Maze II test | expert-normalized score | Improved |
| fine-tune CLIP with VIP and IDM losses | ARP-DT · Procgen test environments | expert-normalized score | Improved |
| fine-tune CLIP representations | ARP-RSSM · RLBench Pick Up Cup test | success rate: 20.37 -> 50.93 | Improved |
| use frozen CLIP rewards instead of baseline | ARP-RSSM · RLBench Pick Up Cup | success rate | No clear effect |
| train multimodal encoder from scratch instead of pre-trained CLIP (ablation) | ARP-DT+ · Procgen all environments | expert-normalized score | Hurt |
| add VIP loss or IDM loss when fine-tuning CLIP (ablation) | ARP-DT · Procgen | expert-normalized score | Improved |
| add return prediction loss (ablation) | ARP-DT · Procgen | expert-normalized score | Improved |
| vary return-prediction loss coefficient lambda (ablation) | ARP-DT · CoinRun held-out | expert-normalized score | No clear effect |
| replace instruction text with random text (ablation) | ARP-DT · Procgen train and eval | expert-normalized score | Hurt |
| use unseen instruction with new object | ARP-DT · CoinRun-bluegem | expert-normalized score | Improved |
| use multimodal reward to distinguish distractors | ARP-DT · Maze III | expert-normalized score | Improved |
| use natural language instruction instead of goal image | ARP-DT vs GC-InstructRL · Procgen | expert-normalized score | Improved |
Closest papers by meaning
- EvIL: Evolution Strategies for Generalisable Imitation Learning (ICML 2024 · similarity 0.88)
- GROOT-2: Weakly Supervised Multimodal Instruction Following Agents (ICLR 2025 · similarity 0.87)
- Solving New Tasks by Adapting Internet Video Knowledge (ICLR 2025 · similarity 0.86)
- Imitation Learning from Observation with Automatic Discount Scheduling (ICLR 2024 · similarity 0.86)
- Imitation Learning from a Single Temporally Misaligned Video (ICML 2025 · similarity 0.86)
- Deep Demonstration Tracing: Learning Generalizable Imitator Policy for Runtime Imitation from a Single Demonstration (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
