Research map / Large language models / Agents, code and math
SorryDB: Can AI Provers Complete Real-World Lean Theorems?
Austin Letson, Leopoldo Sarra, Auguste Poiroux, Oliver Dressler, Paul Lezeau, Dhyan Aranha, Frederick Pu, Aaron Hill, Miguel Corredera Hidalgo, Julian Berman, George Tsoukalas, Lenny Taelman
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add compiler feedback self-correction (16 iterations) | Claude Opus 4.5 · SorryDB-2601 | success rate: 7.8 % -> 27.1 % | Improved |
| add compiler feedback self-correction (16 iterations) | Gemini Flash 3 · SorryDB-2601 | success rate: 10.8 % -> 27.9 % | Improved |
| add LeanSearch tool agent on top of self-correction | Gemini Flash 3 · SorryDB-2601 | success rate: 27.9% -> 30.3% | Improved |
| increase samples from pass@1 to pass@32 | Gemini Flash 3 · SorryDB-2601 | success rate: 10.8 % -> 20.5 % | Improved |
| use general-purpose LLM instead of specialized prover | Goedel Prover V2 32B vs Gemini Flash 3 · SorryDB-2601 | pass@1: 10.8 % -> 2.7% | Hurt |
| combine all provers | all provers · SorryDB-2601 | success rate: 30.3 % -> 35.7 % | Improved |
| use deterministic tactics instead of LLM for some tasks | Tactics · SorryDB-2601 | success rate: 8.4 % | Mixed |
Closest papers by meaning
- APOLLO: Automated LLM and Lean Collaboration for Advanced Formal Reasoning (NeurIPS 2025 · similarity 0.88)
- Lean Workbook: A large-scale Lean problem set formalized from natural language math problems (NeurIPS 2024 · similarity 0.88)
- ImProver: Agent-Based Automated Proof Optimization (ICLR 2025 · similarity 0.88)
- MA-LoT: Model-Collaboration Lean-based Long Chain-of-Thought Reasoning enhances Formal Theorem Proving (ICML 2025 · similarity 0.88)
- Numina-Lean-Agent: An Open and General Agentic Reasoning System for Formal Mathematics (ICML 2026 · similarity 0.88)
- miniF2F-Lean Revisited: Reviewing Limitations and Charting a Path Forward (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
