Research map / Multi-agent systems and games / LLM agents
Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worlds
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Steve-Eye trains an end-to-end large multimodal model (LLaMA-2 with a visual encoder and VQ-GAN visual tokens) on an 850K Minecraft open-world instruction dataset to give LLM-based embodied agents visual perception, knowledge QA, and skill prediction/planning.
Problem
LLM-based embodied agents in open worlds like Minecraft rely on text-only observations and outputs, which limits their grounding in visual scenes, their ability to answer knowledge questions with visual outputs, and their skill feedback and planning.
Approach
Couple a pre-trained LLM with a visual encoder and visual tokenizer to form an end-to-end multimodal I/O model, trained via two-stage instruction tuning on a semi-automatically built 850K Minecraft instruction dataset covering perception, foundational knowledge, and skill interaction.
Key results
- Steve-Eye-13b with CLIP encoder reaches 92.5 inventory and 82.8 equipment accuracy on ENV-VC, outperforming BLIP-2 (41.6 and 58.5).
- On FK-QA TEXT, Steve-Eye-13b achieves 44.36% accuracy versus 41.78% for gpt-turbo-3.5, and 65.13% on FK-QA IMG recipe image generation.
- On 24 skill planning tasks, Steve-Eye achieves higher mean success rates than MineAgent and GPT Assistant across tasks (e.g., 0.53 vs 0.00 for MineAgent on chest).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CLIP instead of MineCLIP as visual encoder | Steve-Eye-13b · ENV-VC test set: inventory prediction | inventory: +48.9 | Helped |
| remove instruction tuning (ablation) | Steve-Eye · ENV-VC test set | inventory: 22.7 -> 92.5 | Hurt |
| remove snapshot description instructions (ablation) | Steve-Eye · ENV-VC test set | inventory: 46.2 (+23.5) -> 92.5 (+69.8) | Hurt |
| remove icon images (ablation) | Steve-Eye · ENV-VC test set | inventory: 52.3 (+29.6) -> 92.5 (+69.8) | Hurt |
| replace rule-based game judgment with model skill prediction (ablation) | Steve-Eye skill planner · Skill planning benchmark: chest task | success rate: 0.53 -> 0.23 | Hurt |
Surprising findings
- CLIP as visual encoder outperformed MineCLIP by large margins on fine-grained inventory and equipment prediction, attributed to MineCLIP lacking fine-grained alignment.
- Llama-2-13b scored lower than 7b on the FK-QA scoring metric, which the authors attribute to question difficulty variation.
Limitations
- Skill prediction is not fully reliable, so it cannot yet replace rule-based game judgment; struggles with fine-grained multi-frame reasoning.
- Visual generation struggles with fine-grained or semantically overlapping recipe images.
- Evaluation limited to Minecraft; generalization to real-world or other environments is untested.
Experimental setup
Datasets: MineDojo snapshots Minecraft Wiki Minecraft recipes ChatGPT-generated instructions
Benchmarks: ENV-VC FK-QA Skill-Pred Skill planning (24 Minecraft tasks)
Metrics: accuracy recall success rate ChatGPT score (0-10)
Baselines: BLIP-2 Llama-2-7b Llama-2-13b Llama-2-70b gpt-turbo-3.5 MineAgent GPT Assistant
Builds on: MineDojo MineCLIP Voyager DEPS Plan4MC LLaMA-2 VQ-GAN LoRA
Closest papers by meaning
- Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld (CVPR 2024 · similarity 0.90)
- From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessons (CVPR 2025 · similarity 0.89)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.89)
- VirtualEnv: A Platform for Embodied AI Research (AAAI 2026 · similarity 0.89)
- VisualAgentBench: Towards Large Multimodal Models as Visual Foundation Agents (ICLR 2025 · similarity 0.89)
- GenRL: Multimodal-foundation world models for generalization in embodied agents (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.