Research map / Multi-agent systems and games / LLM agents
EVMbench: Evaluating AI Agents on Smart Contract Security
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces EVMbench, a benchmark of 117 curated smart contract vulnerabilities from 40 audits that evaluates AI agents on detecting, patching, and exploiting them, with programmatic on-chain grading for Patch and Exploit.
Problem
Existing AI cybersecurity benchmarks do not cover the full lifecycle of smart contract vulnerabilities (discovery, mitigation, exploitation) with realistic, verifiable on-chain grading, despite high financial stakes of smart contracts.
Approach
Builds three audit-derived evaluation modes (Detect via judge-graded audit reports, Patch via test and exploit-test grading, Exploit via replaying agent transactions on a local Anvil chain and checking balances/state) on Code4rena high-severity vulnerabilities, with a Rust harness (ploit/veto) for reproducible, cheat-resistant grading.
Key results
- GPT-5.3-Codex scores highest on Patch (41.7%) and Exploit (71.0%) when run via Codex CLI.
- Claude Opus 4.6 scores highest on Detect at 45.9%.
- With medium mechanism hints, GPT-5.2 reaches 90.2% on Patch and 78.3% on Exploit, suggesting vulnerability discovery is the main bottleneck.
Surprising findings
- Hints sharply raise Patch and Exploit performance, indicating discovery in large repos rather than repair or transaction construction is the primary bottleneck.
- All models earn far less than the attainable Detect audit payout, and a small number of vulnerabilities make up most of the total award mass.
Limitations
- Evaluated vulnerabilities are curated from known audit findings and are not representative of all real-world bugs.
- Detect mode cannot credit valid issues outside the historical report and does not validate extra findings, so false positives go unpenalized.
- Exploit mode runs on a blank local Anvil chain with mock contracts and no forking or cross-chain support, and cannot reproduce complex timing behavior.
Experimental setup
Datasets: Code4rena audit reports Tempo audit vulnerabilities
Benchmarks: EVMbench Detect (117 vulns) EVMbench Patch (44 vulns) EVMbench Exploit (23 vulns)
Metrics: recall/score rate award payout ($) patch success rate exploit success rate
Baselines: OpenCode scaffold (OC-GPT-5.2) OpenAI o3 GPT-5 Claude Opus 4.5 Gemini 3 Pro SCONE-Bench
Builds on: BountyBench OpenCode ReAct
Closest papers by meaning
- BountyBench: Dollar Impact of AI Agent Attackers and Defenders on Real-World Cybersecurity Systems (NeurIPS 2025 · similarity 0.85)
- Agora: Toward Autonomous Bug Detection in Production-Level Consensus Protocols with LLM Agents (ICML 2026 · similarity 0.84)
- MTVHunter: Smart Contracts Vulnerability Detection Based on Multi-Teacher Knowledge Translation (AAAI 2025 · similarity 0.84)
- SCDBench: A Benchmark for LLM-Based Smart Contract Decompilers (NeurIPS 2026 · similarity 0.83)
- SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks (NeurIPS 2025 · similarity 0.83)
- GitInject: Real-World Prompt Injection Attacks in AI-Powered CI/CD Pipelines (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.