Research map / Large language models / Language, safety and interpretability
MineObserver 2.0: A Deep Learning & In-Game Framework for Assessing Natural Language Descriptions of Minecraft Imagery
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Presents MineObserver 2.0, an in-game AI system that captions Minecraft screenshots with an attention-based CNN-LSTM and uses RoBERTa similarity and keyword feedback to assess learner observations, evaluated in a double-blind summer camp study.
Problem
Assessing the accuracy of learners' natural-language observations of Minecraft science-learning imagery in real time, and giving useful feedback to students and teachers.
Approach
Combines an in-game screenshot "Photographer" client, a trained CNN-LSTM attention captioner that produces an expert caption, RoBERTa cosine similarity with keyword extraction for threshold-based feedback, and a teacher-facing Visualizer dashboard, improving over the prior MineObserver.
Key results
- Student ratings of AI-generated captions were higher than MineObserver (Welch t = 1.8896, df = 56.806, p = 0.03196).
- Student ratings of feedback usefulness were higher than MineObserver (t = 3.0733, df = 49.521, p = 0.001719).
- Caption metrics: BLEU-1 0.2 vs 0.197 and METEOR 0.14647 vs 0.12165 for MineObserver; BLEU-3 was higher for MineObserver (0.0117 vs 0.0095).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace MineObserver with MineObserver 2.0 | AI-generated Minecraft image captions · Summer camp students | Mean student accuracy rating: p-value 0.03196 | Helped |
| Replace MineObserver with MineObserver 2.0 | AI feedback · Summer camp students | Mean student usefulness rating: p-value 0.001719 | Helped |
| Replace MineObserver with MineObserver 2.0 | Minecraft image captioning · Minecraft image captioning | BLEU-1: 0.197 -> 0.2 | Helped |
| Replace MineObserver with MineObserver 2.0 | Minecraft image captioning · Minecraft image captioning | BLEU-2: 0.0294 -> 0.033 | Helped |
| Replace MineObserver with MineObserver 2.0 | Minecraft image captioning · Minecraft image captioning | BLEU-3: 0.0117 -> 0.0095 | Hurt |
| Replace MineObserver with MineObserver 2.0 | Minecraft image captioning · Minecraft image captioning | BLEU-4: 0 -> 0.0055 | Helped |
| Replace MineObserver with MineObserver 2.0 | Minecraft image captioning · Minecraft image captioning | METEOR: 0.12165 -> 0.14647 | Helped |
Surprising findings
- MineObserver scored higher than the new model on BLEU-3 (0.0117 vs 0.0095), and BLEU-4 was near zero for both systems.
Limitations
- Small study with 73 students across 6 camps and n=61 ratings; student ratings are subjective Likert scores.
- Captioning metrics show mixed results and very low BLEU-4 scores.
- In-game speed rated median 3 of 5, suggesting room for improvement in latency.
Experimental setup
Datasets: WHIMC Minecraft screenshots with learner and crowdsourced captions (over 2000 datapoints) STSb NLI
Benchmarks: BLEU-1 BLEU-2 BLEU-3 BLEU-4 METEOR student ratings of caption accuracy and feedback usefulness
Metrics: BLEU METEOR Welch t-test on 1-5 Likert ratings in-game speed rating
Baselines: MineObserver
Builds on: MineObserver RoBERTa ResNet Adam
Closest papers by meaning
- ADAM: An Embodied Causal Agent in Open-World Environments (ICLR 2025 · similarity 0.85)
- OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents (NeurIPS 2024 · similarity 0.85)
- CrafterDojo: A Suite of Foundation Models for Building Open-Ended Embodied Agents in Crafter (NeurIPS 2026 · similarity 0.84)
- MrSteve: Instruction-Following Agents in Minecraft with What-Where-When Memory (ICLR 2025 · similarity 0.84)
- MCU: An Evaluation Framework for Open-Ended Game Agents (ICML 2025 · similarity 0.83)
- Experience-based Knowledge Correction for Robust Planning in Minecraft (ICLR 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.