Research map / Multi-agent systems and games / LLM agents
Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Synapse is an LLM computer agent that abstracts raw states, prompts with full successful trajectories (trajectory-as-exemplar), and retrieves exemplars from an embedding memory, reaching 99.2% on 64 MiniWoB++ tasks and large Step SR gains on Mind2Web.
Problem
In-context computer control agents are limited by LLM context length with complex HTML states, by exemplar structures (plans, MCQs) that fail to capture full trajectories in long-horizon tasks, and by hard-coded task-specific exemplars that generalize poorly to novel tasks.
Approach
Three components: few-shot state abstraction to shrink raw states into task-relevant observations, trajectory-as-exemplar prompting that interleaves full abstracted observations and actions, and an embedding-based exemplar memory retrieved by similarity over task metadata.
Key results
- Achieves 99.2% average success rate across 64 MiniWoB++ tasks using demonstrations from only 48 tasks, without self-correction.
- Solves the book-flight task, which prior ICL methods could not handle.
- On Mind2Web with GPT-3.5, improves average Step SR over MindAct by 32%, 50%, and 56% as components are added; with CodeLlama-7B reaches about 2.5x MindAct's Step SR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use state abstraction, TaE prompting, and exemplar memory instead of MindAct prompting | GPT-3.5 on Mind2Web · Mind2Web across Cross-Task, Cross-Website, and Cross-Domain | Step SR: 32%, 50%, and 56% | Helped |
| Use state abstraction with direct generation and fewer top-ranked elements instead of MCQ with top-50 elements (ablation) | Mind2Web · Mind2Web | success rate: 32% | Helped |
| Add state abstraction (ablation) | Synapse on MiniWoB++ email-inbox-nl-turk · email-inbox-nl-turk | success rate: 52% to 100% | Helped |
| Add state abstraction (ablation) | Synapse on MiniWoB++ complex-state tasks · book-flight | success rate | Helped |
| Add TaE prompting to state abstraction (ablation) | GPT-3.5 on Mind2Web · Mind2Web | Step SR: 19% | Helped |
| Use TaE prompting rather than the RCI prompting approach (ablation) | MiniWoB++ long-horizon tasks · guess-number, use-spinner, use-autocomplete | success rate: 20%, 88%, and 58% -> 100%, 100%, and 98% | Helped |
| Add exemplar memory to state abstraction and TaE (ablation) | GPT-3.5 on Mind2Web · Cross-Task | Step SR: 29.2 -> 30.6 | Helped |
| Add exemplar memory to state abstraction and TaE (ablation) | GPT-3.5 on Mind2Web · Cross-Website | Step SR: 22.7 -> 24.2 | Helped |
| Add exemplar memory to state abstraction and TaE (ablation) | GPT-3.5 on Mind2Web · Cross-Domain | Step SR: 26.2 -> 26.4 | Hurt |
| Add exemplar memory to state abstraction and TaE (ablation) | CodeLlama-7B on Mind2Web · Cross-Domain | Step SR: 19.9 -> 19.7 | Hurt |
| Use exemplar memory to retrieve demonstrations from 48 tasks (ablation) | Synapse on MiniWoB++ · MiniWoB++ | success rate: 64 tasks with demonstrations from only 48 tasks | Helped |
Surprising findings
- Reducing top-k elements from 50 to 5 lowered element recall from 86% to 53% yet still raised Mind2Web step success rate.
- Exemplar memory gave negligible gains in cross-domain generalization (0.2 for GPT, -0.2 for CodeLlama).
Limitations
- High inference latency due to reliance on LLMs.
- Dependence on exemplar quality.
- Currently text-only; no multimodal or video understanding.
Experimental setup
Datasets: MiniWoB++ Mind2Web
Benchmarks: MiniWoB++ Mind2Web
Metrics: success rate step success rate element selection accuracy
Baselines: RCI AdaPlanner MindAct CC-Net Pix2Act WebGUM WebN-T5
Builds on: RCI MindAct Chain-of-thought
Closest papers by meaning
- TaskBench: Benchmarking Large Language Models for Task Automation (NeurIPS 2024 · similarity 0.86)
- SYNAPSE: SYmbolic Neural-Aided Preference Synthesis Engine (AAAI 2025 · similarity 0.85)
- StrategyLLM: Large Language Models as Strategy Generators, Executors, Optimizers, and Evaluators for Problem Solving (NeurIPS 2024 · similarity 0.85)
- ReCAP: Recursive Context-Aware Reasoning and Planning for Large Language Model Agents (NeurIPS 2025 · similarity 0.85)
- Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks (NeurIPS 2025 · similarity 0.85)
- Improving Knowledge Extraction from LLMs for Task Learning through Agent Analysis (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.