Research map / Multi-agent systems and games / LLM agents
Agentick: A Unified Benchmark for General Sequential Decision-Making Agents
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Agentick, a 37-task gridworld benchmark with five observation modalities, oracle policies, SFT datasets and a harness, evaluating RL, frozen LLM/VLM, and SFT agents on a unified oracle-normalized score.
Problem
No unified benchmark enables fair comparison of RL agents, foundation-model agents, and hybrids on the same sequential decision-making tasks with aligned semantics and metrics.
Approach
A procedurally generated, Gymnasium-compatible benchmark with 37 tasks in six capability categories, five synchronized observation modalities (ASCII, language, structured, isometric pixels, state arrays), oracle-normalized scoring (ONS) calibrated by programmatic oracle policies, and oracle SFT datasets for training.
Key results
- SFT-250k leads overall at 0.447 ONS, ahead of frozen GPT-5 mini (0.309) and PPO 2M (0.287).
- Reasoner harness improves Qwen3.5-4B ASCII ONS from 0.023 to 0.228 (3-10x across models).
- LoRA SFT on 250k oracle examples lifts Qwen3.5-4B direct Markovian ONS from 0.023 to 0.447.
Surprising findings
- ASCII observations consistently outperform natural language descriptions for LLM agents.
- Qwen3.5-0.8B with reasoning harness (0.094) outperforms the larger Qwen3-4B (0.085).
Limitations
- Gridworld abstraction: discrete, 2D, turn-based tasks do not cover continuous or real-time control.
- No VLM, hybrid RL/FM, or flagship frontier model evaluations in this version.
- Cross-paradigm comparisons are not compute-, supervision-, or modality-matched; oracles not globally optimal on stochastic tasks.
Experimental setup
Datasets: Agentick oracle trajectories (120K, 250K, 500K episodes)
Benchmarks: Agentick
Metrics: Oracle-Normalized Score (ONS) success rate 95% bootstrap CI
Baselines: PPO GPT-5 mini Gemini 3.1 Flash Lite Claude Haiku 4.5 Qwen3.5-4B Qwen3-4B Qwen3.5-2B Qwen3.5-0.8B
Builds on: BALROG MiniGrid PPO Stable-Baselines3
Closest papers by meaning
- FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics (NeurIPS 2026 · similarity 0.89)
- Agent-as-a-Judge: Evaluate Agents with Agents (ICML 2025 · similarity 0.89)
- AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning (ICML 2026 · similarity 0.88)
- AOrchestra: Automating Sub-Agent Creation for Agentic Orchestration (ICML 2026 · similarity 0.88)
- ExACT: Teaching AI Agents to Explore with Reflective-MCTS and Exploratory Learning (ICLR 2025 · similarity 0.88)
- UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks (NeurIPS 2026 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.