Research map / Large language models / Agents, code and math
Language Models Meet World Models: Embodied Experiences Enhance Language Models
Jiannan Xiang, Tianhua Tao, Yi Gu, Tianmin Shu, Zirui Wang, Zichao Yang, Zhiting Hu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| finetune on embodied experiences from VirtualHome with EWC-LoRA | GPT-J-6B · Object Path Tracking | LCS: 33.86 -> 98.67 | Improved |
| finetune with E2WM (EWC-LoRA) | GPT-J-6B · Counting | accuracy: 30.41 -> 67.01 | Improved |
| finetune with E2WM (EWC-LoRA) | GPT-J-6B · Plan Gen | Rouge-L: 34.31 -> 51.23 | Improved |
| finetune with E2WM (EWC-LoRA) | GPT-J-6B · Pile test subset | perplexity: 3.443 -> 3.537 | No clear effect |
| add EWC regularization to LoRA (ablation) | LoRA finetuning · Pile test subset (GPT-Neo) | perplexity: 4.360 -> 4.193 | Improved |
| use LoRA instead of full-parameter EWC (ablation) | EWC · Pile test subset (GPT-Neo) | perplexity: 4.995 -> 4.360 | Improved |
| replace EWC with KL penalty (ablation) | LoRA finetuning · Pile test subset (GPT-Neo) | perplexity: 4.193 -> 5.029 | Hurt |
| add EWC to LoRA (ablation) | LoRA finetuning · Counting (GPT-Neo) | accuracy: 27.84 -> 28.87 | No clear effect |
| add plan generation training task (ablation) | E2WM GPT-Neo · Plan Gen | Rouge-L: 14.48 -> 49.70 | Improved |
| add activity recognition training task (ablation) | E2WM GPT-Neo · Act Recog | accuracy: 48.63 -> 85.43 | Improved |
| add object path tracking training task (ablation) | E2WM GPT-Neo · Obj PT | score: 29.90 -> 85.91 | Improved |
| add counting training task (ablation) | E2WM GPT-Neo · Counting | accuracy: 35.05 -> 28.87 | Hurt |
Closest papers by meaning
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.91)
- World Model Implanting for Test-time Adaptation of Embodied Agents (ICML 2025 · similarity 0.90)
- ModularAgent: A Task-Aware Modular Framework for Joint Optimization of Multimodal Large Language Models and World Models (CVPR 2026 · similarity 0.90)
- Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making (NeurIPS 2024 · similarity 0.89)
- Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld (CVPR 2024 · similarity 0.89)
- From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
