Research map / Reinforcement learning / Visual and representation RL

Open-World Reinforcement Learning over Long Short-Term Imagination

ICLR 2025 · Reinforcement learning › Visual and representation RL · approach: Scalable RL systems and environments · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes LS-Imagine, a model-based visual RL agent that adds goal-conditioned jumpy state transitions and affordance maps (via image zoom-in scored by MineCLIP) to DreamerV3-style world models, improving exploration in MineDojo.

Problem

Visual model-based RL agents in high-dimensional open-world environments like Minecraft are short-sighted because they optimize policies over short imagined rollouts (about 15 steps), which hampers exploration for long-horizon payoffs.

Approach

Build a long short-term world model that simulates jumpy transitions toward task-relevant future states, with a jump flag switching between one-step and jumpy imagination. Targets come from affordance maps computed by sliding-window zoom-ins scored by MineCLIP, which serve as goal-conditioned priors and an intrinsic reward.

Key results

• On harvest log in plains, LS-Imagine reaches 80.63% success in 503 steps versus 53.33% and 711 steps for DreamerV3.
• On shear sheep, LS-Imagine achieves 54.28% success versus 25.13% for DreamerV3 and 21.54% for PTGM.
• On mine iron ore, LS-Imagine gets 20.28% success versus 16.79% for DreamerV3, with more steps (1748.55 vs 1789.06 for DreamerV3, only marginally fewer).

Measured effects reported in the paper

Surprising findings

• Series (sequential) long short-term imagination outperforms parallel imagination by large margins.
• Stochastically skipping the jump (30% of the time) keeps gradient feedback on short-term actions.

Limitations

• Computational overhead introduced by the long short-term design and affordance map generation.
• Validated only in 3D navigation Minecraft environments with embodied agents; not suited to fixed-viewpoint or 2D settings.
• Affordance maps may fail for targets MineCLIP has not encountered.

Experimental setup

Datasets: MineDojo tasks random-agent collected images annotated with MineCLIP affordance maps

Benchmarks: MineDojo harvest log in plains MineDojo harvest water with bucket MineDojo harvest sand MineDojo shear sheep MineDojo mine iron ore

Metrics: success rate steps per successful episode MineCLIP value

Baselines: DreamerV3 VPT STEVE-1 PTGM Director DECKARD

Builds on: DreamerV3 MineCLIP Swin-Unet MineDojo

Closest papers by meaning

• Zero-shot World Models via Search in Memory (NeurIPS 2025 · similarity 0.89)
• Reinforcing VLAs in Task-Agnostic World Models (NeurIPS 2026 · similarity 0.89)
• Dynamics-Aligned Latent Imagination in Contextual World Models for Zero-Shot Generalization (NeurIPS 2025 · similarity 0.88)
• RIG: Synergizing Reasoning and Imagination in End-to-End Generalist Policy (ICLR 2026 · similarity 0.87)
• Acting Beyond Learning: Imagination-Assisted Decision-Making in the Visual-based Multi-Agent Cooperative Scenarios (AAAI 2025 · similarity 0.87)
• XLand-100B: A Large-Scale Multi-Task Dataset for In-Context Reinforcement Learning (ICLR 2025 · similarity 0.87)