Research map / Vision-language and multimodal / Robot action and navigation
OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OmniJARVIS, a Minecraft vision-language-action model that tokenizes behavior trajectories with a self-supervised FSQ behavior tokenizer and trains an MLM on unified interaction-data token sequences.
Problem
Open-world instruction-following agents struggle when VLA models communicate with controllers only via text goals, or when they emit low-level control commands directly, which is inefficient for long-horizon tasks.
Approach
Learn a self-supervised behavior encoder/policy decoder with FSQ quantization to produce discrete behavior tokens, add them to a pretrained multimodal LM's vocabulary, and model interleaved instruction, memory, observation, thought, and behavior-token sequences autoregressively so the same model reasons and acts.
Key results
- On long-horizon programmatic tasks OmniJARVIS achieves 0.59 average success rate versus 0.43 for DEPS and 0.02 for GROOT.
- On open-ended embodied QA OmniJARVIS scores 8.40 versus 7.50 for GPT-3.5 and 2.34 for Vicuna-7B.
- On open-ended instruction following (FSD, lower is better) OmniJARVIS gets 886.2 versus 929.5 for DEPS and 975.9 for VPT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use unified behavior-token modeling instead of the DEPS language-goal planner | OmniJARVIS on Minecraft programmatic tasks · Programmatic tasks (average) | success rate: $0.43$ -> $0.59$ | Helped |
| Use unified behavior-token modeling instead of the DEPS language-goal planner | OmniJARVIS on Minecraft open-ended instruction following · Open-ended instruction following | Instruction Following: $929.5$ -> $886.2$ | Helped |
| Use OmniJARVIS rather than GPT-3.5 | Minecraft open-ended question answering · Open-ended question answering | QA: $7.50$ -> $8.40$ | Helped |
| Increase the behavior-token codebook from e8 to e14 (ablation) | OmniJARVIS on Minecraft · Collect wood | Average Rewards: ${0.63^{\pm0.67}}$ -> $2.27^{\pm2.45}$ | Helped |
| Increase the behavior-token codebook from e10 to e14 (ablation) | OmniJARVIS on Minecraft · Collect wood | Average Rewards: ${0.54^{\pm1.21}}$ -> $2.27^{\pm2.45}$ | Helped |
| Increase the behavior-token codebook from e8 to e14 (ablation) | OmniJARVIS behavior tokenizer · Behavior-token reconstruction | Reconstruction FSD: $46.57$ -> $42.72$ | Helped |
| Increase the behavior-token codebook from e10 to e14 (ablation) | OmniJARVIS behavior tokenizer · Behavior-token reconstruction | Reconstruction FSD: $43.67$ -> $42.72$ | Helped |
| Increase the behavior-token codebook from e8 to e14 (ablation) | OmniJARVIS behavior tokenizer · Behavior-token sampling | Sampling FSD: $68.90$ -> $57.37$ | Helped |
| Increase the behavior-token codebook from e10 to e14 (ablation) | OmniJARVIS behavior tokenizer · Behavior-token sampling | Sampling FSD: $61.85$ -> $57.37$ | Helped |
| Replace FSQ with VQ-VAE for behavior tokenization (ablation) | OmniJARVIS behavior tokenizer · Minecraft programmatic tasks | success rate | Hurt |
| Replace FSQ behavior tokens with sub-goal language annotations (ablation) | OmniJARVIS behavior tokenizer · Minecraft programmatic tasks | success rate | Hurt |
| Train without instructions on the trajectories (ablation) | OmniJARVIS interaction data · Behavior-token prediction | eval loss | Hurt |
Surprising findings
- VQ-VAE behavior tokenizer collapsed during training, yielding no results.
- Language-goal behavior tokens performed worse than FSQ behavior tokens in ablations.
Limitations
- Tokenizer and policy decoder trained on a limited set of Minecraft contractor data; generalization beyond Minecraft shown only in a single Atari experiment.
- Thoughts, memories and instructions are synthesized by GPT-3.5, so data quality depends on LLM labeling.
- Evaluation on open-ended tasks relies on FSD and a custom embodied QA benchmark with limited size (over 100 questions).
Experimental setup
Datasets: OpenAI Contractor Minecraft dataset (VPT) mc-eqa-300k (generated from Minecraft Wiki) JARVIS-1 agent recordings Montezuma's Revenge RND-agent dataset (1.8M transitions)
Benchmarks: atomic tasks (chopping trees, digging dirt, mining stones, collecting wheat seeds) programmatic tasks (30 tasks in five groups) open-ended instruction following with FSD embodied QA benchmark
Metrics: average reward success rate Fréchet Sequence Distance (FSD) accuracy (LLM-as-judge score) eval loss codebook usage
Baselines: VPT STEVE-I GROOT Open-world Control GPT ReAct DEPS Voyager Vicuna-7B Vicuna-13B LLaMA2-70B GPT-3.5
Builds on: GROOT LLaVA FSQ VPT MineCLIP
Closest papers by meaning
- ADAM: An Embodied Causal Agent in Open-World Environments (ICLR 2025 · similarity 0.89)
- Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worlds (ICLR 2024 · similarity 0.88)
- GROOT: Learning to Follow Instructions by Watching Gameplay Videos (ICLR 2024 · similarity 0.87)
- Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks (NeurIPS 2024 · similarity 0.87)
- MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception (CVPR 2024 · similarity 0.87)
- ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Prompting (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.