Research map / Multi-agent systems and games / LLM agents

Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Agentic benchmarks and orchestration · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Claw-Eval, a 300-task agent benchmark with trajectory-level auditing (traces, audit logs, snapshots), integrated Completion/Safety/Robustness scoring with error injection, and Average/Pass@k/Pass^k metrics, evaluated on 14 frontier models.

Problem

Existing agent benchmarks grade only final outputs, embed safety and robustness poorly, and cover few modalities and interaction paradigms, so they cannot reliably measure deployable agent capability.

Approach

Record each run through three independent evidence channels (execution traces, service-side audit logs, environment snapshots), embed safety constraints in ordinary workflow tasks as a multiplicative gate, measure robustness via controlled error injection, and report Pass@k alongside Pass^k to separate capability ceiling from reliability.

Key results

• A vanilla LLM judge misses 44% of safety violations (12/27) and 13% of robustness issues (15/118) detected by the hybrid pipeline.
• Under error injection up to 0.6, Pass@3 stays nearly flat while Pass^3 drops by up to 24 percentage points.
• Claude-Opus-4.6 leads overall Pass^3 at 70.4%, and the best Multimodal Pass^3 is only 25.7% (GPT-5.4).

Surprising findings

• Round count in multi-turn dialogue is nearly uncorrelated with performance (r=0.07), while question precision explains 76% of Pass^3 variance.
• Gemini-3.1-Pro ranks 2nd on multi-turn Pass^3 but only 7th on General Pass^3, showing model rankings shift across task groups.

Limitations

• Robustness evaluation covers only infrastructure-level transient failures (HTTP 429/500, latency), not semantic or adversarial perturbations.
• Open-ended rubric items rely on LLM judges, with about 95% agreement with human scoring.
• Public benchmark may be optimized against over time.

Experimental setup

Datasets: OfficeQA PinchBench FinanceQA OneMillion-Bench Video-MME LongVideoBench

Benchmarks: Claw-Eval

Metrics: Average Score Pass@3 Pass^3 Completion Safety Robustness

Baselines: Claude-Opus-4.6 Claude-Sonnet-4.6 GPT-5.4 Gemini-3.1-Pro Gemini-3-Flash Qwen3.5-397B-A17B MiMo-V2-Pro MiMo-V2-Omni GLM-5-Turbo GLM-5V-Turbo DeepSeek-V3.2 MiniMax-M2.7

Builds on: tau-bench PinchBench OSWorld LLM-as-a-judge

Closest papers by meaning

• AgentHijack: Benchmarking Computer Use Agent Robustness to Common Environment Corruptions (ICML 2026 · similarity 0.90)
• OpenAgentSafety: A Comprehensive Framework For Evaluating Real-World AI Agent Safety (ICLR 2026 · similarity 0.89)
• Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis (ICLR 2026 · similarity 0.89)
• Towards a Science of AI Agent Reliability (ICML 2026 · similarity 0.89)
• UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks (NeurIPS 2026 · similarity 0.89)
• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox (ICML 2026 · similarity 0.89)