Research map / Multi-agent systems and games / Human-AI and goal-conditioned
Learning to Model the World With Language
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Dynalang, a multimodal world-model agent that learns from diverse language (hints, manuals, instructions) by predicting future text and image representations, then acts via imagined rollouts in DreamerV3.
Problem
Agents usually map only task instructions to actions via language-conditioned policies, which degrade as language becomes more diverse (descriptions, dynamics, corrections); the paper asks whether learning language-conditioned world models lets agents use such diverse language.
Approach
Treat language as an observation modality in a multimodal world model trained with future-representation prediction (plus image, language, reward, continuation reconstruction), decoupling supervised world-model learning from RL policy learning via latent imagination; text-only pretraining is possible by zeroing image and action inputs.
Key results
- On HomeGrid, Dynalang improves with more language hint types while language-conditioned IMPALA and R2D2 degrade with more diverse language.
- On Messenger, Dynalang fits the hardest stage S3 to non-trivial performance where EMMA, IMPALA and R2D2 fail to fit it at all.
- On VLN-CE, Dynalang succeeds at significantly more instructions than model-free R2D2 but remains behind state-of-the-art VLN methods that use expert demonstrations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| provide diverse language hints in addition to task instructions | Dynalang · HomeGrid | task score | Helped |
| provide more diverse language hints | R2D2 · HomeGrid | task score | Hurt |
| use a language-conditioned world model rather than a language-conditioned policy | Dynalang · Messenger S1–S3 | game performance | Helped |
| use Dynalang rather than R2D2 for instruction following | VLN-CE · VLN-CE | success rate | Helped |
| regularize language actions toward the world model's predicted next token (ablation) | Dynalang · LangRoom, vocabulary size 10,000 | question-answer accuracy | Helped |
| scale the language vocabulary to 10,000 without a world-model prior (ablation) | Dynalang · LangRoom | question-answer accuracy | Hurt |
| pretrain on TinyStories text | Dynalang · Messenger S2 | task performance | Helped |
| pretrain on in-domain Messenger manuals | Dynalang · Messenger S2 | task performance | Helped |
| replace one-hot token encodings with pretrained embeddings (ablation) | Dynalang · LangRoom S2 | question-answer accuracy | Helped |
| pretrain Dynalang with a small amount of text-only data | Dynalang with one-hot token encodings · LangRoom S2 | question-answer accuracy | Helped |
| replace token-by-token language input with sentence embeddings (ablation) | DreamerV3 language-conditioning alternatives · Messenger S1 | game performance | Hurt |
| use a language-conditioned policy instead of joint language-image representations in the world model (ablation) | DreamerV3 · Messenger S1 | game performance | Hurt |
Surprising findings
- Language-conditioned model-free R2D2 performance drops as it receives more diverse language hints in HomeGrid.
- Pretraining on TinyStories exceeds the final performance of using pretrained T5 embeddings on Messenger S2.
Limitations
- VLN-CE performance is not competitive with state-of-the-art VLN methods that use expert demonstrations or specialized architectures.
- Language hints are scripted with templates in controlled environments rather than natural human language at scale.
- Experiments use 2-3 seeds and small-scale environments; text-only pretraining is only shown at small scale.
Experimental setup
Datasets: HomeGrid (procedurally generated) Messenger VLN-CE / Matterport3D LangRoom (procedurally generated) TinyStories
Benchmarks: HomeGrid Messenger VLN-CE LangRoom
Metrics: task score / reward success rate QA accuracy
Baselines: IMPALA R2D2 EMMA Language-Conditioned Policy Sentence Embed T5 with Image Adapter T5 with Cross-Attention R2D2 (VLN-CE)
Builds on: DreamerV3 Recurrent State Space Model T5 IMPALA R2D2
Closest papers by meaning
- Reinforcement World Model Learning for LLM-based Agents (NeurIPS 2026 · similarity 0.88)
- Do LLMs Build World Representations? Probing Through the Lens of State Abstraction (NeurIPS 2024 · similarity 0.88)
- GenRL: Multimodal-foundation world models for generalization in embodied agents (NeurIPS 2024 · similarity 0.87)
- Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation (ICLR 2025 · similarity 0.87)
- World–Value–Action Model: Implicit Planning for Vision–Language–Action Systems (NeurIPS 2026 · similarity 0.87)
- Language Control Diffusion: Efficiently Scaling through Space, Time, and Tasks (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.