Research map / Large language models / Agents, code and math
Faults in Our Formal Benchmarking: Dataset Defects and Evaluation Failures in Lean Theorem Proving
Pawan Sasanka Ammanamanchi, Siddharth Bhat, Stella Biderman
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| manually correct flawed formal statements (20 problems with proven issues) | DeepSeek-Prover-V2-7B · 20 corrected benchmark problems | problems solved: 0/20 -> 3/20 | Improved |
| manually correct flawed formal statements (20 problems with proven issues) | Kimina-Prover-8B · 20 corrected benchmark problems | problems solved: 0/20 -> 2/20 | Improved |
| repair weakened statements (ProofNet to ProofNet#) | provers · ProofNet | pass rate | Hurt |
| add LLM false-positive filtering to static checkers | static checkers · ProverBench | number of findings: 427 -> 277 | Improved |
| use Gemini 3.0 Flash instead of DeepSeek-V3 for warning filtering | LLM verifier · ProverBench warning set | accuracy: 68.5% -> 83.3% | Improved |
| LLM semantic audit on specification errors versus formalization errors | Sonnet 4.5 + Thinking · 92-problem challenge set | F1: 0.51 vs 0.18 | Mixed |
| use Sonnet 4.5 + Thinking instead of GPT-5.2 + Thinking for semantic audit | semantic audit · 92-problem challenge set | accuracy: 54.6% -> 68.1% | Improved |
Closest papers by meaning
- miniF2F-Lean Revisited: Reviewing Limitations and Charting a Path Forward (NeurIPS 2025 · similarity 0.89)
- CLEVER: A Curated Benchmark for Formally Verified Code Generation (NeurIPS 2025 · similarity 0.88)
- MA-LoT: Model-Collaboration Lean-based Long Chain-of-Thought Reasoning enhances Formal Theorem Proving (ICML 2025 · similarity 0.88)
- FaithSieve: Fine-Grained Evaluation of Math Proofs with Faithful Formal Evidence (NeurIPS 2026 · similarity 0.88)
- Lean Workbook: A large-scale Lean problem set formalized from natural language math problems (NeurIPS 2024 · similarity 0.87)
- APOLLO: Automated LLM and Lean Collaboration for Advanced Formal Reasoning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
