Research map / Continual learning / Online and replay-based
Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CL-Bench, six expert-validated stateful tasks with a normalized gain metric, showing that naive in-context learning beats dedicated memory systems for continual learning in LLM agents.
Problem
Existing benchmarks do not measure whether LLM-based systems genuinely improve from sequential experience by discovering latent, task-specific structure shared across instances.
Approach
Tasks are built so that shared latent structure (codebase layout, schema conventions, opponent strategies) is learnable online but not recoverable from pretraining; a gain metric compares stateful reward with the same system run statelessly on the same instance, and a concept-drift variant tests adaptation.
Key results
- ICL with Claude Sonnet 4.6 achieves the highest normalized gain (25.4%) and normalized reward (22.3%) among evaluated systems.
- ICL Notepad with Sonnet 4.6 ranks sixth by gain (18.2%) with normalized reward of only 3.5%.
- ACE with GPT-5.4 costs $62.8 per run yet reaches only 8.6% normalized gain.
Surprising findings
- Naive full-context ICL outperforms dedicated memory systems such as Mem0 and ACE.
- Some systems show virtually no stable learning, with gains driven by repeated exposure to the same variant offset by information loss at switches.
Limitations
- Task sequences are on the order of tens of instances, shorter than real deployment horizons.
- Only context-based memory paradigms evaluated; parametric test-time training not covered.
- Tasks require frontier-level capability, so smaller model failure modes are not surfaced.
Experimental setup
Datasets: SWE-bench Amazon Products Metadata and Reviews
Benchmarks: CL-Bench
Metrics: normalized reward normalized gain stability plasticity cost (USD)
Baselines: ICL Notepad Mem0 ACE Claude Code Codex
Builds on: Mem0 ACE SWE-bench TerminalBench
Closest papers by meaning
- Continual Reinforcement Learning by Planning with Online World Models (ICML 2025 · similarity 0.87)
- EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective (NeurIPS 2026 · similarity 0.87)
- Learning to Continually Learn via Meta-learning Agentic Memory Designs (NeurIPS 2026 · similarity 0.87)
- StreamBench: Towards Benchmarking Continuous Improvement of Language Agents (NeurIPS 2024 · similarity 0.87)
- AgentOdyssey: Open-Ended Long-Horizon Text Game Generation for Test-Time Continual Learning Agents (NeurIPS 2026 · similarity 0.87)
- AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.