Research map / Continual learning / Online and replay-based

Autonomous Continual Learning for Environment Adaptation of Computer-Use Agents

NeurIPS 2026 · Continual learning › Online and replay-based · approach: Plasticity-preserving continual reinforcement learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes ACuRL, an autonomous curriculum RL framework where a GPT-5 task generator synthesizes environment-grounded tasks from exploration and web-crawled context, with a GRPO-trained CUA and a CUAJudge evaluator, enabling zero-human-data continual adaptation of computer-use agents.

Problem

Computer-use agents encounter unseen and evolving digital environments, and continual adaptation requires high-quality environment-grounded training tasks and reliable trajectory-level rewards without costly human annotation.

Approach

Grounds task generation in environment exploration and web-crawled contexts, adapts task difficulty per iteration from measured agent success rates (easy: add complexity, medium: diversify, hard: decompose), and uses CUAJudge, a WebJudge extension with state-difference analysis and evidence-grounded key point verification, as the binary reward for multi-turn GRPO.

Key results

• Intra-environment continual learning yields 3–29% absolute gains on target environments without forgetting elsewhere (overall gains of 2–7%).
• CUAJudge reaches 87.5% agreement with the rule-based evaluator on 1,444 OSWorld trajectories, versus 85.6% for WebJudge, and 93.7% agreement with human judgments on 288 RL trajectories.
• Cross-environment sequential learning raises UI-TARS-1.5-7B overall score from 19.5 to 25.9 and Qwen3-VL-8B from 22.0 to 31.7 across six environments.

Surprising findings

• Updates are highly sparse: about 80% of parameters in both the LLM backbone and vision encoder remain unchanged across RL iterations.
• Replacing GPT-5 with the open-source Qwen3-VL-8B as task generator yields only a small drop (30.8 to 29.5 on Impress at iteration 1).

Limitations

• Depends on proprietary GPT-5 and GPT-5-mini for task generation and evaluation.
• UI-TARS-1.5-7B has weak instruction-following, so it cannot serve as its own task generator; fully autonomous self-evolution is not achieved.
• Agent performance remains far below human level, with some environments under 50% success.

Experimental setup

Datasets: OSWorld OfficeWorld ScienceBoard

Benchmarks: OSWorld OfficeWorld ScienceBoard

Metrics: success rate precision recall agreement

Baselines: Base Agent (UI-TARS-1.5-7B / Qwen3-VL-8B) WebJudge SEAgent WebRL DigiRL

Builds on: GRPO WebJudge WebRL SEAgent VeRL

Closest papers by meaning

• SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience (ICML 2026 · similarity 0.87)
• Continual Reinforcement Learning by Planning with Online World Models (ICML 2025 · similarity 0.86)
• Learn-by-interact: A Data-Centric Framework For Self-Adaptive Agents in Realistic Environments (ICLR 2025 · similarity 0.86)
• Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments (NeurIPS 2026 · similarity 0.86)
• Online Continual Learning for Interactive Instruction Following Agents (ICLR 2024 · similarity 0.86)
• Learning to Continually Learn via Meta-learning Agentic Memory Designs (NeurIPS 2026 · similarity 0.86)