Research map / Multi-agent systems and games / LLM agents

Verify0: Can AI Agents Build Formally Verified Software Repositories?

NeurIPS 2026 · Multi-agent systems and games › LLM agents · approach: Software engineering agent benchmarks

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Vero, a 43-instance repository-level benchmark in Lean 4 for joint implementation and proof synthesis from Python/Dafny/Verus/Coq sources, with a formal audit mechanism, and evaluates frontier coding agents showing the best solves 27/43.

Problem

Whether AI agents can jointly write implementations and machine-checked proofs across real multi-module verified repositories; existing benchmarks cover single functions or proof-only tasks with fixed implementations.

Approach

A repository-level Lean 4 benchmark with predetermined API interfaces and specifications parameterized over the implementation record, supporting proof-only and code-and-proof modes, plus an audit mechanism where agents formally prove spec unsatisfiability or reference-code incorrectness to surface benchmark errors.

Key results

• GPT-5.5 (xhigh) fully solves 27 of 43 instances in code-and-proof mode and 25 in proof-only mode.
• Ten instances resist all eight configurations across both modes.
• GPT-5.5 (xhigh) passes 87.3% of specifications in code-and-proof mode and 85.8% in proof-only mode.

Surprising findings

• Agents sometimes replace hard-to-verify reference algorithms with simpler implementations that satisfy the same specifications, gaining provability.
• Agents rarely revisit implementations when proofs get stuck, growing proof text instead of refactoring code.

Limitations

• Targets Lean 4 only.
• Corpus favors code that translates cleanly into Lean; concurrent or temporal protocols are absent.
• Audit mechanism certifies satisfiability but cannot ensure specifications are semantically correct or complete.

Experimental setup

Datasets: Verus/Dafny/Coq source repositories (Track 1) Python repositories (Track 2)

Benchmarks: Vero

Metrics: fully solved instances specification pass rate

Baselines: Claude Opus 4.8 Claude Sonnet 5 GPT-5.5 (medium)

Builds on: Lean 4 Mathlib Dafny Verus Coq

Closest papers by meaning

• NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents (ICML 2026 · similarity 0.89)
• AlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinement (ICML 2025 · similarity 0.89)
• Commit0: Library Generation from Scratch (ICLR 2025 · similarity 0.88)
• APE-Bench: Evaluating Automated Proof Engineering for Formal Math Libraries (ICML 2026 · similarity 0.87)
• Build-and-Find: An Effort-Aware Protocol for Evaluating Agent-Managed Codebases (NeurIPS 2026 · similarity 0.87)
• From I/O to Code with Discovery Agent (NeurIPS 2026 · similarity 0.87)