Research map / Multi-agent systems and games / LLM agents

ContractBench: Can LLM Agents Preserve Observation Contracts?

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Agent safety and attack benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces ContractBench, a deterministic benchmark of 33 tasks testing whether tool-using LLM agents preserve observation contracts (temporal validity and byte-level integrity of tool-returned artifacts), evaluating 38 model variants.

Problem

LLM agents often fail real API workflows by using tool-returned artifacts (presigned URLs, tokens, OAuth state) after expiry or with altered bytes, but existing agent benchmarks do not test these inter-step constraints on both validity and integrity axes.

Approach

Formalizes observation contracts as tool-returned artifacts with a time-to-live and a SHA-256 integrity predicate, and builds a dual-axis, programmatically validated benchmark (virtual clock, HTTP-log validators, 15-label failure taxonomy) to measure compliance across many models.

Key results

• Claude Opus 4.6 leads at 77.8% success rate on ContractBench (77/99); no evaluated model clears 80%.
• Qwen3.5 shows a within-family cliff from 0% at 4B to 56.6% at 9B, reaching 70.7% at 397B-A17B.
• Failure-label-guided retry on 42 GPT-5.1 failures gives 19.0% pass with correct labels vs 11.9% with wrong labels (+7.1 pp).

Surprising findings

• GPT-5 family shows a non-monotonic V-shape: GPT-5.1 regresses to 48.5% from GPT-5's 70.7% before GPT-5.2 recovers to 74.8%.
• DeepSeek-R1 (671B MoE) scores 0%, suggesting chain-of-thought training does not improve observation-contract compliance.

Limitations

• Virtual clock abstracts away real network latency and API variability.
• No human baseline is provided.
• Frontier proprietary models have undisclosed parameter counts, limiting scaling analysis.

Experimental setup

Datasets: ContractBench

Benchmarks: ContractBench

Metrics: success rate per-task pass rate failure-label distribution

Baselines: GPT-5 GPT-5.1 GPT-5.2 GPT-4o Gemini 2.5 Pro Gemini 2.5 Flash Claude Sonnet 4.5

Builds on: Harbor Terminal-Bench TicToc Reflexion Self-Refine

Closest papers by meaning

• LLM Agents Already Know When to Call Tools - Even Without Reasoning (NeurIPS 2026 · similarity 0.88)
• ComplexMCP: Evaluation of LLM Agents in Dynamic, Interdependent, and Large-Scale Tool Sandbox (ICML 2026 · similarity 0.88)
• TAI3: Testing Agent Integrity in Interpreting User Intent (NeurIPS 2025 · similarity 0.87)
• Why Do Multi-Agent LLM Systems Fail? (NeurIPS 2025 · similarity 0.87)
• MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Use (ICLR 2026 · similarity 0.87)
• SecureClaw: Clawing Back Control of LLM Agents (NeurIPS 2026 · similarity 0.87)