Research map / Multi-agent systems and games / LLM agents
SE-GA: Memory-Augmented Self-Evolution for GUI Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SE-GA, a GUI agent combining hierarchical test-time memory (episodic, semantic, experiential) with a two-stage SFT-plus-GRPO self-evolution training pipeline on Qwen2.5-VL-7B.
Problem
GUI agents struggle with long-horizon multi-step tasks due to limited context windows and static policies that do not adapt or reuse past successful experiences.
Approach
Test-Time Memory Extension retrieves episodic, semantic and experiential memories at inference, and Memory-Augmented Self-Evolution trains the policy with SFT on grounding data then GRPO (with DAPO-style token-level ratios, adaptive cosine clipping, and hierarchical rewards) on self-collected trajectories relabeled via Hindsight Goal-Shifting.
Key results
- SE-GA reaches 89.0 average on ScreenSpot, outperforming 7B baselines and UI-TARS-72B (88.4).
- SE-GA achieves 75.8% success rate on AndroidControl-High (main table), and 83.9% step success on GUIOdyssey.
- On AndroidWorld SE-GA reports 39.0% versus UI-TARS-7B at 33.0%.
Surprising findings
- Removing TTME lowers AndroidControl-High SR from 73.8 to 61.4, and removing MASE drops GUIOdyssey SR from 83.9 to 60.4.
- Success-only experiential retrieval underperforms Top-k and Mixed retrieval, suggesting failed trajectories provide useful signal.
Limitations
- Memory retrieval via embedding and visual similarity may add significant inference latency as the repository grows.
- Training data is small (about 4K trajectories) and the authors note scaling as future work.
- Inconsistencies in reported numbers: AndroidControl-High SR is 75.8 in the main table but 73.8 in the ablation table, and baselines are largely copied from UI-TARS rather than re-run.
Experimental setup
Datasets: AITW AMEX GUIOdyssey ScreenSpot AndroidControl AndroidWorld
Benchmarks: ScreenSpot AndroidControl-Low AndroidControl-High GUIOdyssey AndroidWorld
Metrics: action type accuracy grounding accuracy success rate
Baselines: UI-TARS OS-Atlas Aguvis Qwen2.5-VL GPT-4o GUI-Critic-R1 OS-Genesis SeeClick
Builds on: GRPO DAPO hindsight experience replay Qwen2.5-VL
Closest papers by meaning
- Darwinian Memory: A Training-Free Self-Regulating Memory System for GUI Agent Evolution (ICML 2026 · similarity 0.92)
- Executable Agentic Memory for GUI Agent (ICML 2026 · similarity 0.89)
- SE-GUI: Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning (NeurIPS 2025 · similarity 0.89)
- Co-EPG: A Framework for Co-Evolution of Planning and Grounding in Autonomous GUI Agents (AAAI 2026 · similarity 0.87)
- MemEvolve: Meta-Evolution of Agent Memory Systems (ICML 2026 · similarity 0.87)
- Agent-SAMA: State-Aware Mobile Assistant (AAAI 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.