Research map / Large language models / Agents, code and math
Improving Knowledge Extraction from LLMs for Task Learning through Agent Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes STARS, which wraps an LLM in an embodied cognitive agent via beam-search response trees, agent-side analysis and repair of viability mismatches, and LLM-based selection, to elicit viable goal descriptions for one-shot task learning in simulation.
Problem
LLM responses used as goal knowledge for embodied agents often are uninterpretable, ungrounded, or infeasible for the agent's embodiment, and prompt engineering alone requires heavy human oversight to correct them.
Approach
Generate a broad space of LLM responses via beam search, have the agent mentally simulate and categorize mismatches (language, grounding, affordance), re-prompt with targeted repair templates, and select among viable candidates by asking the LLM to choose, so users only need to confirm or reject a vetted option.
Key results
- STARS achieves 77.5% (kitchen), 94.4% (groceries), 92.9% (office) task completion without oversight, versus 52.5%, 66.7%, 35.7% for TBP.
- With oversight, STARS+O reaches 100% completion on all tasks and reduces user words for kitchen from 403 to 127.
- Over 70% of LLM responses for tidy kitchen were not viable; only 13% were situationally relevant.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace TBP with STARS | Embodied task-learning agent · Tidy kitchen | task completion (%): 52.5 -> 77.5 | Helped |
| Replace TBP with STARS | Embodied task-learning agent · Store groceries | task completion (%): 66.7 -> 94.4 | Helped |
| Replace TBP with STARS | Embodied task-learning agent · Organize office | task completion (%): 35.7 -> 92.9 | Helped |
| Add human oversight | STARS · Tidy kitchen | task completion (%): 77.5 -> 100.0 | Helped |
| Add human oversight | STARS · Store groceries | task completion (%): 94.4 -> 100.0 | Helped |
| Add human oversight | STARS · Organize office | task completion (%): 92.9 -> 100.0 | Helped |
| Use beam Search Tree without Analysis and Repair or Selection (ablation) | STARS · Tidy kitchen | task completion (%): 77.5 -> 50.0 | Hurt |
| Add Analysis and Repair to beam Search Tree (ablation) | STARS · Tidy kitchen | task completion (%): 50.0 -> 77.5 | Helped |
| Add LLM Selection to beam Search Tree, without Analysis and Repair (ablation) | STARS · Tidy kitchen | task completion (%): 50.0 -> 40.0 | Hurt |
| Add LLM Selection to beam Search Tree, without Analysis and Repair (ablation) | STARS · Store groceries | task completion (%): 66.7 -> 66.7 | No clear effect |
| Add LLM Selection to beam Search Tree, without Analysis and Repair (ablation) | STARS · Organize office | task completion (%): 21.4 -> 21.4 | No clear effect |
| Add LLM Selection to beam Search Tree with Analysis and Repair (ablation) | STARS · Store groceries | task completion (%): 77.8 -> 94.4 | Helped |
Surprising findings
- Selection via LLM did not improve over mean log-probability choice for the tidy kitchen task due to missing context; adding context raised it to 92.5%.
- Search Tree alone gave no improvement over TBP; its benefit came only when combined with analysis and repair.
Limitations
- Experiments are in a simulated environment with a limited set of objects and tasks.
- Selection does not consistently beat log-probability ranking without additional context.
- Token cost grows substantially due to recursive beam search and repair.
Experimental setup
Benchmarks: simulated tidy kitchen task (35 objects) store groceries task (15 objects) organize office task (12 objects)
Metrics: task completion rate percentage of viable and situationally relevant responses number of user words and instructions token counts
Baselines: TBP TBP+O ST STS STAR STARS+O
Builds on: template-based prompting self-consistency interactive task learning Inner Monologue Recursively Criticizes and Improves
Closest papers by meaning
- Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks (NeurIPS 2025 · similarity 0.88)
- Do LLMs ``know'' internally when they follow instructions? (ICLR 2025 · similarity 0.87)
- AutoManual: Constructing Instruction Manuals by LLM Agents via Interactive Environmental Learning (NeurIPS 2024 · similarity 0.87)
- SkillOpt: Executive Strategy for Self-Evolving Agent Skills (NeurIPS 2026 · similarity 0.87)
- SkillMaster: Toward Autonomous Skill Mastery in LLM Agents (NeurIPS 2026 · similarity 0.87)
- TaskBench: Benchmarking Large Language Models for Task Automation (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.