Research map / Reinforcement learning / Imitation learning
ELEMENTAL: Interactive Learning from Demonstrations and Vision-Language Models for Reward Design in Robotics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ELEMENTAL, which combines a VLM (GPT-4o) drafting feature functions from visual demonstrations with approximate MaxEnt-IRL for reward weights and a self-reflection loop that refines features.
Problem
LLM-based reward design struggles to weight reward features, generalizes poorly out-of-distribution, and cannot capture user intent from text-only task descriptions in robotic RL.
Approach
Use a VLM to draft interpretable feature functions from visual keyframe or superimposed demonstrations, then learn feature weights via an approximate MaxEnt-IRL loop (PPO policy interleaved with gradient ascent on weights) and iteratively refine features by feeding feature-count discrepancies back to the VLM.
Key results
- ELEMENTAL achieves on average 42.3% higher task performance than EUREKA and outperforms it on eight of nine IsaacGym tasks.
- On Ant out-of-distribution variants, ELEMENTAL averages 41.3% better generalization than EUREKA.
- About 80% of ELEMENTAL's first-iteration code executes successfully versus under 50% for EUREKA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ELEMENTAL instead of EUREKA | IsaacGym benchmark tasks · Nine IsaacGym tasks | task success: 42.3% higher performance | Helped |
| Use ELEMENTAL instead of EUREKA | Ant task variants · Ant variants | task success: 41.3% | Helped |
| Add self-reflection (ablation) | ELEMENTAL · IsaacGym tasks | task success: 233.92 -> 114.66 (Cartpole); 464.40 -> 153.52 (Ball Balance) | Helped |
| Add gradient normalization (ablation) | ELEMENTAL · IsaacGym tasks | task success: 22.97 -> 13.39 (Allegro Hand); 2.71 -> 2.32 (Shadow Hand) | Helped |
| Add weight normalization (ablation) | ELEMENTAL · IsaacGym tasks | task success: 8.49 -> 3.23 (Ant); 4.70 -> 4.87 (Humanoid) | Mixed |
| Add visual demonstration input (ablation) | ELEMENTAL · IsaacGym tasks | task success: 464.40 -> 304.58 (Ball Balance); 22.97 -> 18.52 (Allegro Hand) | Helped |
| Use visual demonstrations instead of text state-vector demonstrations (ablation) | ELEMENTAL · IsaacGym tasks | task success: 0.36 vs 0.00 (Franka Cabinet); 22.97 vs 7.07 (Allegro Hand) | Helped |
| Use high-quality instead of random visual demonstrations (ablation) | ELEMENTAL · IsaacGym tasks | task success: 233.92 vs 269.46 (Cartpole); 464.40 vs 352.53 (Ball Balance); 8.49 vs 7.13 (Ant) | Mixed |
| Use visual input (ablation) | ELEMENTAL · Ant with lighter gravity | task success: 5.89 vs 3.14 | Helped |
Surprising findings
- Using random visual demonstrations still yields nontrivial performance on some tasks (e.g., Cartpole 269.46 vs ELEMENTAL 233.92), showing task-level variance in reliance on demonstration content.
- EUREKA with GPT-4o failed to produce any executable reward for ShadowHand, so its improvement there was set to 100% by convention.
Limitations
- Relies on environment rollouts for IRL updates, increasing wall-clock runtime versus EUREKA.
- Performance degrades with suboptimal or random demonstrations.
- Results depend on GPT-4o and use best-of-three-seeds reporting to account for LLM variability.
Experimental setup
Datasets: IsaacGym tasks demonstrations (5 per task, from RL-trained policies)
Benchmarks: IsaacGym Cartpole BallBalance Quadcopter FrankaCabinet Ant Humanoid Anymal AllegroHand ShadowHand Ant variants (reversed obs, lighter gravity, running backward)
Metrics: average task success over final 100 training steps reward correlation with ground-truth reward code execution rate
Baselines: EUREKA Behavior Cloning IRL Random Policy GT Reward
Builds on: EUREKA MaxEnt-IRL AIRL PPO
Closest papers by meaning
- Text2Reward: Reward Shaping with Language Models for Reinforcement Learning (ICLR 2024 · similarity 0.89)
- GoalLadder: Incremental Goal Discovery with Vision-Language Models (NeurIPS 2025 · similarity 0.89)
- Learning Reward for Robot Skills Using Large Language Models via Self-Alignment (ICML 2024 · similarity 0.88)
- Eureka: Human-Level Reward Design via Coding Large Language Models (ICLR 2024 · similarity 0.88)
- Multi-Stage Manipulation with Demonstration-Augmented Reward, Policy, and World Model Learning (ICML 2025 · similarity 0.88)
- Efficient Language-instructed Skill Acquisition via Reward-Policy Co-Evolution (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.