Research map / Multi-agent systems and games / LLM agents

XSkill: Continual Learning from Experience and Skills in Multimodal Agents

ICML 2026 · Multi-agent systems and games › LLM agents · approach: Self-evolving language agent learning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes XSkill, a training-free dual-stream (skills + experiences) continual-learning framework for multimodal tool-using agents, with visually grounded knowledge extraction and context-aware retrieval/adaptation, evaluated on five benchmarks with four proprietary backbones.

Problem

Multimodal agents use tools inefficiently and orchestrate them inflexibly, and existing learning-from-experience methods are text-only, so they fail to ground retrieved knowledge in visual observations and cannot continually improve without parameter updates.

Approach

Maintains two complementary knowledge stores: Markdown skill documents (task-level workflows and code templates) and a JSON experience bank (action-level condition-action tips). Knowledge is extracted from multi-path rollouts using visually grounded summarization and cross-rollout critique, consolidated hierarchically, and at inference retrieved via task decomposition, rewritten to the current ima

Key results

• On TIR-Bench with Gemini-3-Flash, XSkill reaches 47.75% Average@4, 11.13 points above the strongest baseline Agent-KB.
• Across four backbones, XSkill improves Average@4 by 2.58 to 6.71 points over the tool-only baseline.
• Ablation on VisualToolBench with Gemini-2.5-Pro: removing experiences drops Average@4 by 3.04 and removing skills by 3.85 points.

Surprising findings

• With Qwen3-VL-235B on VisualToolBench, XSkill lowers Average@4 versus the tool-only baseline (11.52 vs 11.80), suggesting transferred knowledge can interfere with weaker models' native tool use.
• Agent-KB beats XSkill on VisualToolBench Average@4 for Qwen3-VL-235B (12.85 vs 11.52), and XSkill is not best on all settings.

Limitations

• Gains are mixed for open-source Qwen3-VL models, where transferred knowledge can hurt Average@4 on VisualToolBench.
• Evaluation covers only a single accumulation-then-test cycle; iterative continual refinement is not empirically tested.
• Experiments use a small training set of 100 tasks per benchmark and rely on proprietary backbones.

Experimental setup

Datasets: VisualToolBench TIR-Bench MMSearch-Plus AgentVista

Benchmarks: VisualToolBench TIR-Bench MMSearch-Plus AgentVista MMBrowseComp

Metrics: Average@4 Pass@4

Baselines: w/ Tools (no learning) Agent Workflow Memory Dynamic CheatSheet Agent-KB No Tools

Builds on: Agent Workflow Memory Dynamic CheatSheet Agent-KB

Closest papers by meaning

• ViLoMem: Agentic Learner with Grow-and-Refine Multimodal Semantic Memory (CVPR 2026 · similarity 0.90)
• AgentVista: Evaluating Multimodal Agent in Ultra-Challenging Realistic Visual Scenarios (NeurIPS 2026 · similarity 0.88)
• DeepEyesV2: Toward Agentic Multimodal Model (ICLR 2026 · similarity 0.88)
• Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks (ICLR 2026 · similarity 0.88)
• Optimus-1: Hybrid Multimodal Memory Empowered Agents Excel in Long-Horizon Tasks (NeurIPS 2024 · similarity 0.88)
• Agentic Jigsaw Interaction Learning for Enhancing Visual Perception and Reasoning in Vision-Language Models (ICLR 2026 · similarity 0.87)