Research map / Reinforcement learning / Visual and representation RL

MHWA: Multi-timescale Hierarchical World-Action Model

NeurIPS 2026 · Reinforcement learning › Visual and representation RL · approach: RL for combinatorial routing

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MHWA, a two-level world-action model with a strided, gradient-isolated high-level context manager and a context-conditioned MoE low-level worker, evaluated on a 14-game offline Atari subset against JOWA.

Problem

Autoregressive world models accumulate open-loop rollout drift, and existing hierarchical world models add complexity and interference without reliably beating flat baselines.

Approach

A fixed token-stride high-level manager trained only by its own skip-step objective (stop-gradient to low-level losses) supplies slow context to a low-level Transformer whose FFN layers are sparse MoE gated on both context and current token state.

Key results

• MHWA-77M achieves IQM HNS 0.884 (95% CI [0.736, 1.031]) across three seeds versus 0.529 for a matched JOWA-70M.
• MHWA-77M's IQM HNS is numerically comparable to the published JOWA-150M result (0.833) with 70.88% fewer forward+backward GFLOPs per sequence.
• On the 6-game 44M ablation, removing the stride (k=1) drops IQM by 62.1%, the largest drop among variants.

Surprising findings

• Ablation shows that the fixed-weight manager with stop-gradient supplies useful trajectory-dependent context, since zero or Gaussian replacement of the manager state reduces IQM from 0.503 to 0.406 or 0.381.
• The reported JOWA-70M baseline is trained by the authors with a halved batch and doubled steps relative to published JOWA, and MHWA's IQM is compared against a published JOWA-150M number under different training conditions.

Limitations

• Evaluation is limited to discrete-action offline Atari with a pretrained tokenizer and fixed stride.
• Ablations use a 44M model on a 6-game subset, and the 14-game comparison to JOWA-150M uses published numbers under different batch and step settings.
• Long-horizon open-loop rollouts do not eliminate eventual manifold departure.

Experimental setup

Datasets: RL Unplugged Atari (rlu-atari) DQN Replay Dataset

Benchmarks: Atari 14-game subset (10% offline data) Atari held-out fine-tuning (5 games) 6-game ablation subset

Metrics: IQM HNS Median HNS DQN-normalized score MSE PSNR SSIM

Baselines: JOWA-70M JOWA-150M JOWA-40M MTBC MGDT EDT Scaled-QL

Builds on: JOWA CQL C51 Sparse Mixture-of-Experts

Closest papers by meaning

• Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretraining (ICLR 2025 · similarity 0.87)
• EDELINE: Enhancing Memory in Diffusion-based World Models via Linear-Time Sequence Modeling (NeurIPS 2025 · similarity 0.86)
• WALL-E: World Alignment by NeuroSymbolic Learning improves World Model-based LLM Agents (NeurIPS 2025 · similarity 0.85)
• Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL (NeurIPS 2026 · similarity 0.85)
• From Forecasting to Planning: Policy World Model for Collaborative State-Action Prediction (NeurIPS 2025 · similarity 0.84)
• DyMoDreamer: World Modeling with Dynamic Modulation (NeurIPS 2025 · similarity 0.84)