Research map / Large language models / Agents, code and math
PACEbench: A Framework for Evaluating Practical AI Cyber-Exploitation Capabilities
Zicheng Liu, Lige Huang, Jie Zhang, Dongrui Liu, Yuan Tian, Jing Shao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use PACEagent instead of CAI agent framework | Claude-3.7-Sonnet · PACEbench A-CVE | score: +0.18 | Improved |
| use PACEagent instead of CAI agent framework | Claude-3.7-Sonnet · PACEbench B-CVE | score: +0.05 | Improved |
| use PACEagent instead of CAI agent framework | Claude-3.7-Sonnet · PACEbench C-CVE | score: +0.20 | Improved |
| use PACEagent instead of CAI agent framework | Claude-3.7-Sonnet · PACEbench overall | score: 65.2% performance gain | Improved |
| PACEagent multi-stage design vs CAI token use | Claude-3.7-Sonnet · PACEbench | token consumption: 28% more tokens | Hurt |
| add benign hosts to environment (B-CVE vs A-CVE) | PACEagent · PACEbench | score: Claude-3.7-Sonnet 0.412 -> 0.263 | Hurt |
| add multi-host dependencies (C-CVE vs B-CVE) | PACEagent · PACEbench | score: Claude-3.7-Sonnet 0.263 -> 0.267 | Hurt |
| add WAF defenses (D-CVE) | PACEagent · PACEbench | score: 0.241 -> 0.000 | Hurt |
| increase vulnerability difficulty (lower human pass rate) | PACEagent · PACEbench A-CVE | number of successful models | Hurt |
| use open-source models with small context windows instead of proprietary ones | PACEagent · PACEbench | score: Qwen3-32B 0.024, Deepseek-V3 0.012 vs Claude-3.7-Sonnet 0.241 | Hurt |
| use Claude 4 instead of Claude 3.7 | PACEagent · PACEbench preliminary | success rate | Hurt |
Closest papers by meaning
- Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models (ICLR 2025 · similarity 0.91)
- CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale (ICLR 2026 · similarity 0.89)
- CVE-Bench: A Benchmark for AI Agents’ Ability to Exploit Real-World Web Application Vulnerabilities (ICML 2025 · similarity 0.88)
- Cyber-Zero: Training Cybersecurity Agents without Runtime (ICLR 2026 · similarity 0.88)
- PACE: A Proxy for Agentic Capability Evaluation (NeurIPS 2026 · similarity 0.87)
- A New Framework for Cybersecurity Refusals in AI Agents (ICML 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
