Research map / Multi-agent systems and games / LLM agents

OpenClawBench: Benchmarking Process-side Anomalies in Real-world Agent Execution Trajectories

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces OpenClawBench, a 31k-trajectory dataset of real OpenClaw agent runs on BFCL tasks with FullTax silver process-anomaly labels, and shows a LoRA-tuned Gemma 3 12B detector beats GPT-5.4 zero-shot on binary F1.

Problem

Task success can hide process-side anomalies (unresolved ambiguity, unsafe writes, ignored errors, weak grounding, capability overcommitment) in real agent executions, and no trainable dataset separates task outcome from process reliability.

Approach

Converts real OpenClaw sessions on BFCL tasks into ReAct-style normalized trajectories, fuses them with oracle outcomes, and uses a multi-stage LLM annotation protocol (FullTax) to produce binary anomaly labels, subtypes, evidence, onset localization, severity and recoverability, then trains a small detector on the high-confidence pool.

Key results

• Fine-tuned Gemma 3 12B + LoRA reaches binary F1 0.729 on the 2,646-trajectory test split vs 0.426 for GPT-5.4 zero-shot and 0.372 for the base model.
• 2,904 of 31,135 oracle-passing executions (9.33%) are labeled process-anomalous; 1,765 of 1,904 high-risk oracle-passing executions (92.70%) are anomalous.
• FullTax silver labels match human audit on 288 of 300 trajectories (96.0%).

Surprising findings

• The fine-tuned detector's gain comes from calibration rather than recall: zero-shot detectors over-flag at 42-50% against a 14.7% label rate.
• The detector retains its lift (F1 0.767 vs GPT-5.4 0.483) when gpt-oss-20B trajectories are held out from training.

Limitations

• All FullTax labels are LLM-generated silver labels; the human audit covers only 300 trajectories.
• Headline comparison uses a single fine-tuned backbone against a single frontier reference, so the lift mixes fine-tuning with backbone choice.
• Tail subtypes have fewer than 35 test examples, so macro-F1 is effectively a 3-way score.

Experimental setup

Datasets: BFCL

Benchmarks: OpenClawBench

Metrics: binary F1 precision recall macro-F1 predicted-anomaly rate

Baselines: GPT-5.4 (zero-shot, reasoning) Base Gemma 3 12B (no FT)

Builds on: ReAct BFCL LoRA

Closest papers by meaning

• Automata from Agent Traces: Failure and Next-Step Prediction (NeurIPS 2026 · similarity 0.88)
• CORRECT: COndensed eRror RECognition via knowledge Transfer in multi-agent systems (ICML 2026 · similarity 0.87)
• AgentForesight: Online Auditing for Early Failure Prediction in Multi-Agent Systems (NeurIPS 2026 · similarity 0.87)
• AgentCollabBench: Diagnosing When Good Agents Make Bad Collaborators (NeurIPS 2026 · similarity 0.87)
• From Patches to Trajectories: Privileged Process Supervision for Software-Engineering Agents (NeurIPS 2026 · similarity 0.86)
• Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents (NeurIPS 2026 · similarity 0.86)