Research map / Multi-agent systems and games / LLM agents
VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes In-Context Abstraction Learning (ICAL), where VLM agents turn suboptimal demonstrations into annotated programs of thought via self-reflection and human feedback, then use them for retrieval-augmented prompting or fine-tuning.
Problem
VLM/LLM agents need high-quality exemplar demonstrations in context, but raw human or noisy demonstrations are suboptimal and expert-written examples are costly to produce.
Approach
A frozen VLM abstracts noisy trajectories into optimized action sequences annotated with cognitive abstractions (causal relations, state changes, subgoals, visual task construals), which are refined via natural-language human feedback during execution and stored in a growing memory for retrieval or SFT.
Key results
- On TEACh unseen, ICAL retrieval reaches 35.1% success and 49.3% GC with ground-truth perception, and ICAL with fine-tuning plus retrieval reaches 35.8% success and 54.2% GC; the estimated-perception setting gives 10.5% success vs. HELPER 8.3%.
- On VisualWebArena, ICAL with GPT4V reaches 22.7% average success (vs. 14.3% for GPT4V+SoM), ICAL GPT4o reaches 23.4% (vs. 18.9%), and fine-tuned Qwen2-VL-7B reaches 8.2% (vs. 2.9%).
- On Ego4D action forecasting, ICAL reduces noun edit distance to 0.6934 from 0.7575 for few-shot CoT and gets action distance 0.9242, competitive with supervised 0.9235.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ICAL retrieval examples instead of raw visual demonstrations | TEACh unseen validation, ground-truth perception · TEACh unseen validation | Success: 17.2 -> 35.1 | Helped |
| Add ICAL retrieval examples instead of HELPER hand-written examples | TEACh unseen validation, ground-truth perception · TEACh unseen validation | GC: 36.7 -> 49.3 | Helped |
| Combine fine-tuning with ICAL retrieval | GPT3.5-turbo-1106 on TEACh · TEACh unseen validation | GC: 49.3 -> 54.2 | Helped |
| Remove the programs-of-thought generation phase (ablation) | ICAL on TEACh · TEACh unseen validation | Success: 35.1 -> 29.4 | Hurt |
| Remove the human-in-the-loop phase (ablation) | ICAL on TEACh · TEACh unseen validation | GC: 49.3 -> 41.0 | Hurt |
| Add retrieval re-ranking (ablation) | ICAL on TEACh · TEACh unseen validation | Success: 35.1 -> 35.3 | Helped |
| Use GPT-4 instead of GPT-3.5 (ablation) | ICAL on TEACh · TEACh unseen validation | GC: 49.3 -> 63.6 | Helped |
| Fine-tune Qwen2-VL-7B on ICAL examples | Qwen2-VL-7B · VisualWebArena | Average success: 2.9 -> 8.2 | Helped |
| Use ICAL retrieval with GPT4V instead of GPT4V + Set of Marks | GPT-4V · VisualWebArena | Average success: 14.3 -> 22.7 | Helped |
| Remove image input from ICAL examples (ablation) | ICAL with GPT4V · VisualWebArena | Average success: 22.2 -> 19.0 | Hurt |
| Provide the full text trajectory instead of a single-step image example (ablation) | ICAL with GPT4V · VisualWebArena | Average success: 22.2 -> 25.5 average; 28.0 -> 57.7 seen; 21.6 -> 21.6 unseen | Mixed |
| Use ICAL examples for few-shot action forecasting | GPT-4V · Ego4D unseen validation | Noun edit distance ED@(Z=20): 0.7575 -> 0.6934 | Helped |
Surprising findings
- Full-trajectory textual examples raised seen success on VisualWebArena but did not improve unseen success.
- Using GPT-3.5 instead of GPT-4 for ICAL learning roughly halved the number of successful abstracted tasks (52 vs. 122).
Limitations
- Cannot handle extremely misleading demonstrations or feedback.
- Relies on a fixed action API, which may restrict adaptability in dynamic environments.
- GPT-4V visual grounding deficiencies are not fully overcome by in-context learning.
Experimental setup
Datasets: TEACh VisualWebArena Ego4D
Benchmarks: TEACh VisualWebArena Ego4D
Metrics: task success rate goal-condition success edit distance (verb, noun, action) success rate environment steps human feedback per example
Baselines: HELPER Zero-shot CoT Raw Visual Demos Raw Kinesthetic Demos GPT4V + Set of Marks GPT4o Gemini-Pro-1.5 CogVLM IDEFICS-80B-Instruct Qwen2-VL-7B Few-shot CoT Supervised SlowFast baseline
Builds on: DAgger LoRA HELPER Set-of-Marks prompting chain-of-thought
Closest papers by meaning
- Provoking Multi-modal Few-Shot LVLM via Exploration-Exploitation In-Context Learning (CVPR 2025 · similarity 0.89)
- VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environments (CVPR 2026 · similarity 0.88)
- LMAct: A Benchmark for In-Context Imitation Learning with Long Multimodal Demonstrations (ICML 2025 · similarity 0.88)
- Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks (NeurIPS 2025 · similarity 0.88)
- ViLoMem: Agentic Learner with Grow-and-Refine Multimodal Semantic Memory (CVPR 2026 · similarity 0.88)
- Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMs (CVPR 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.