Research map / Reinforcement learning / RL from feedback
Auto MC-Reward: Automated Dense Reward Design with Large Language Models for Minecraft
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Auto MC-Reward uses GPT-4 agents (Reward Designer, Reward Critic, Trajectory Analyzer) to iteratively write executable dense reward functions for sparse-reward Minecraft tasks, training PPO agents initialized from GITM-data imitation models.
Problem
Minecraft provides only sparse binary rewards, making RL exploration for long-horizon tasks (diamond mining, approaching trees and cows on plains) very inefficient; existing dense reward methods perform poorly there.
Approach
An LLM writes white-box Python dense reward functions from task and game descriptions, a second LLM checks the code for syntax and semantic errors, and a third LLM analyzes failed trajectories to suggest revisions in an iterative loop, with reward scale constrained to a fixed set of values.
Key results
- Auto MC-Reward raises diamond-ore exploration success from 0.5% (sparse RL) and 38.9% (imitation) to 45.2%, with 70.0% lava escape rate.
- On approaching trees in plains, success rate reaches 73.4% vs 65.6% for the best dense baseline (MineCLIP) and 4.3% for sparse RL.
- Full diamond-obtaining pipeline achieves 36.5% success in forest biome (7.7% above imitation learning) and 28.1% in plains, where imitation learning gets 0%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace sparse rewards with LLM-designed dense rewards | RL agent · Explore Diamond Ore Underground | succ. (%): 0.5 -> 45.2 | Helped |
| replace sparse rewards with LLM-designed dense rewards | RL agent · Approach Tree on Plains | succ. (%): 4.3 -> 73.4 | Helped |
| replace sparse rewards with LLM-designed dense rewards | RL agent · Approach Cow on Plains | succ. (%): 12.6% vs. 56.3% | Helped |
| add Reward Critic to Reward Designer (ablation) | Auto MC-Reward · Explore Diamond Ore Underground | Succ.: 35.1% to 40.5% | Helped |
| add Trajectory Analyzer to Reward Designer (ablation) | Auto MC-Reward · Explore Diamond Ore Underground | Succ.: 40.5% to 43.1% | Helped |
| add Trajectory Analyzer to Reward Designer and Reward Critic (ablation) | Auto MC-Reward · Explore Diamond Ore Underground | Succ.: 40.5% to 45.2% | Helped |
| add Reward Critic to Reward Designer and Trajectory Analyzer (ablation) | Auto MC-Reward · Explore Diamond Ore Underground | Succ.: 43.1% vs. 45.2% | Helped |
| use Auto MC-Reward IL + RL instead of GITM-guided imitation learning alone | Diamond obtaining agent · Obtaining Diamond (forest) | Diamond Succ. (%): 28.8% vs. 36.5% | Helped |
Surprising findings
- Excessive lava penalties caused agents to look up or down to avoid lava, halting exploration, which required a further reward revision.
- Sparse-reward RL achieved a low death rate (1.5%) only because agents got stuck in place.
Limitations
- Trajectory analysis limited to the last 32 frames due to LLM context length, hindering analysis of long-term failures.
- Relies on GPT-4 and a pretrained imitation foundation model; results depend on GITM-generated data.
- Reward function requires predefined observation inputs and task descriptions, so it is not fully open-ended.
Experimental setup
Datasets: GITM-generated Minecraft diamond-mining gameplay (about 11M frames, 153 hours) MineDojo harvest-mode environment
Benchmarks: MineDojo explore diamond ore underground approach tree in plains approach cow in plains obtaining diamond tech tree
Metrics: success rate death rate lava escape rate average steps average distance
Baselines: Naive Handcraft Imitation Learning RL with Sparse Reward RL with Curiosity Dense Reward RL with Self-Imitation Dense Reward RL with MineCLIP Dense Reward DreamerV3 DEPS VPT GITM
Builds on: VPT PPO GITM MineDojo MineCLIP
Closest papers by meaning
- DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learning (NeurIPS 2025 · similarity 0.85)
- Text2Reward: Reward Shaping with Language Models for Reinforcement Learning (ICLR 2024 · similarity 0.85)
- RL-GPT: Integrating Reinforcement Learning and Code-as-policy (NeurIPS 2024 · similarity 0.85)
- Dense Reward for Free in Reinforcement Learning from Human Feedback (ICML 2024 · similarity 0.85)
- LARM: Large Auto-Regressive Model for Long-Horizon Embodied Intelligence (ICML 2025 · similarity 0.85)
- Experience-based Knowledge Correction for Robust Planning in Minecraft (ICLR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.