Research map / Large language models / Agents, code and math
AutoBaxBuilder: Bootstrapping Code Security Benchmarking
Tobias von Arx, Niels Mündler, Mark Vero, Maximilian Baader, Martin Vechev
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| generate tests and exploits with AutoBaxBuilder instead of expert writing | BaxBench scenarios · BaxBench (28 scenarios) | pass@1 and model ranking | No clear effect |
| use AutoBaxBuilder exploits instead of human-written exploits | BaxBench scenarios · BaxBench (28 scenarios) | sec_pass@1: more than twice as many solutions marked insecure | Improved |
| generate exploits via contrastive secure/insecure solution refinement (RefineExploit) | AutoBaxBuilder · BaxBench scenarios, manual audit | exploit soundness: 1 unsound of 71 exploits | Improved |
| iterate over multiple reference solutions during exploit refinement | AutoBaxBuilder · BaxBench scenarios | CWE coverage: 6 scenarios (21%) test more CWEs | Improved |
| construct scenarios with a disjoint set of LLMs (ablation) | AutoBaxBench · AutoBaxBench vs alternative 40 scenarios | Spearman rank correlation: greater than 0.9 | No clear effect |
| evaluate construction models on their own generated scenarios versus neutral models (ablation) | AutoBaxBench · AutoBaxBench and alternative 40 scenarios | self-bias | No clear effect |
| increase scenario endpoints (Easy to Hard) in AutoBaxBench | Claude-4.5 Sonnet · AutoBaxBench | sec_pass@1: 36% -> 40% -> 25% (Easy, Medium, Hard) | Hurt |
| add agentic harness to code-generating LLMs (ablation) | LLM code generation · BaxBench scenarios | security | No clear effect |
| include CWE-400 resource exhaustion exploits (ablation) | AutoBaxBuilder and BaxBench exploits · BaxBench scenarios | exploit validity | Hurt |
Closest papers by meaning
- SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks (NeurIPS 2025 · similarity 0.91)
- BaxBench: Can LLMs Generate Correct and Secure Backends? (ICML 2025 · similarity 0.90)
- AlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinement (ICML 2025 · similarity 0.88)
- SecCodePRM: A Process Reward Model for Code Security (ICML 2026 · similarity 0.88)
- ProSec: Fortifying Code LLMs with Proactive Security Alignment (ICML 2025 · similarity 0.87)
- CoRe: Benchmarking LLMs’ Code Reasoning Capabilities through Static Analysis Tasks (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
