Research map / Large language models / Agents, code and math
Infer Human’s Intentions Before Following Natural Language Instructions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes FISER, a framework that makes human intentions explicit intermediate steps (social reasoning to infer the robot's task, optional human plan recognition, then embodied planning) for following ambiguous instructions, evaluated on HandMeThat.
Problem
Natural language instructions from humans are ambiguous because they omit information assumed from the speaker's hidden goals and intentions; standard grounding and planning methods do not model these as partially observable factors.
Approach
Decompose instruction following into social reasoning (predict the human's subgoal and the robot's task from the instruction, observed trajectory and state) and embodied reasoning (goal-conditioned planning), with the Transformer predicting intermediate human-intention outputs at intermediate layers; also prompts GPT-4 Turbo with the same step-wise FISER structure.
Key results
- Transformer+FISER reaches 64.5% average success rate on HandMeThat v2 test set, state-of-the-art on the benchmark.
- Multi-staged Transformer+FISER achieves 89.0% on Level 1, 74.0% on Level 2, 52.3% on Level 3, while GPT-4 Turbo vanilla achieves 72.0%, 16.0%, 5.0%, 9.0%.
- On HandMeThat v1 subset, FISER achieves 89.7% on Level 1 vs 27.7% for prior SOTA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add robot task recognition as an intermediate step instead of directly predicting actions (ablation) | Transformer trained from scratch · HMT Levels 1–4 | success rate (%): 77.7±1.6 -> 89.0±1.5; 55.3±0.4 -> 74.0±0.3; 36.3±1.0 -> 52.3±2.3; 38.3±1.4 -> 42.7±0.2 | Helped |
| Add robot task recognition with CoT prompting (ablation) | GPT-4 Turbo with domain-specific prompt engineering · HMT Levels 2–4 | success rate (%): 25.0 -> 36.0; 13.0 -> 18.0; 9.0 -> 17.0 | Helped |
| Add human plan recognition to FISER (ablation) | Transformer + FISER · HMT Level 1 | success rate (%): 89.0±1.5 -> 72.0±1.5 | Hurt |
| Add human plan recognition to FISER (ablation) | Transformer + FISER · HMT Level 4 | success rate (%): 42.7±0.2 -> 51.0±1.2 | Helped |
| Add human plan recognition to FISER (ablation) | GPT-4 Turbo + FISER · HMT Levels 1–4 | success rate (%): 80.0 -> 77.0; 36.0 -> 34.0; 18.0 -> 17.0; 17.0 -> 20.0 | Mixed |
| Train FISER modules multi-staged rather than end-to-end (ablation) | Transformer + FISER · HMT Levels 1–4 | success rate (%): 74.3±0.2 -> 89.0±1.5; 73.7±0.5 -> 74.0±0.3; 47.3±2.1 -> 52.3±2.3; 41.3±1.9 -> 42.7±0.2 | Helped |
| Train FISER+PR modules multi-staged rather than end-to-end (ablation) | Transformer + FISER + PR · HMT Levels 1–4 | success rate (%): 61.0±1.5 -> 72.0±1.5; 66.7±1.1 -> 74.0±0.9; 47.0±1.2 -> 52.3±1.0; 42.0±0.3 -> 51.0±1.2 | Mixed |
| Add domain-specific prompt engineering (ablation) | GPT-4 Turbo · HMT Levels 1–3 | success rate (%): 72.0 -> 82.0; 16.0 -> 25.0; 5.0 -> 13.0 | Helped |
| Train a small Transformer from scratch with FISER rather than prompt GPT-4 Turbo | HMT instruction-following models · HMT Levels 2–4 | success rate (%): 36.0 -> 74.0; 18.0 -> 52.3; 17.0 -> 42.7 | Helped |
Surprising findings
- Training a small (~5M parameter) Transformer from scratch outperforms CoT-prompted GPT-4 Turbo by a large margin on ambiguous levels.
- Adding human plan recognition hurts Level 1 performance in Transformer models while helping Level 4.
Limitations
- Evaluation restricted to a single trial within 4-5 steps, dropping the original step-count-sensitive metric.
- Prior SOTA comparison done on a subset of HandMeThat v1 due to incompatible versions.
- GPT-4 prompting results are based on a single prompt design and limited by irrelevant-object distractors.
Experimental setup
Datasets: HandMeThat version 2 HandMeThat version 1 (subset for comparison)
Benchmarks: HandMeThat
Metrics: success rate QSVO prediction accuracy Obj prediction accuracy
Baselines: Human Heuristic Seq2Seq GPT-4 Turbo vanilla GPT-4 Turbo+PE GPT-4 Turbo+FISER prior SOTA on HandMeThat v1 (iterated goal inference)
Builds on: GreaseLM Chain-of-Thought prompting HandMeThat
Closest papers by meaning
- ThinkBot: Embodied Instruction Following with Thought Chain Reasoning (ICLR 2025 · similarity 0.88)
- Implicit Intelligence - Evaluating Agents on What Users Don’t Say (ICML 2026 · similarity 0.88)
- MetaMind: Modeling Human Social Thoughts with Metacognitive Multi-Agent Systems (NeurIPS 2025 · similarity 0.88)
- Theory of Mind and Persuasion Beyond Conversation: Assessing the Capacity of LLMs to Induce Belief States via Planning and Action (NeurIPS 2026 · similarity 0.87)
- PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks (ICLR 2025 · similarity 0.86)
- EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.