Research map / Large language models / Agents, code and math
FormalML: A Benchmark for Evaluating Formal Subgoal Completion in Machine Learning Theory
Xiao-Wen Yang, Zihao Zhang, Jianuo Cao, Zhi Zhou, Zenan Li, Lan-Zhe Guo, Yuan Yao, Taolue Chen, Yu-Feng Li, Xiaoxing Ma
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add 10 candidate premises to the prompt (ablation) | DeepSeek-Prover-V2 (noCoT) · FormalML retrieval subset | Pass@32 (%): 58.37 -> 69.29 | Improved |
| add 10 candidate premises to the prompt (ablation) | DeepSeek-Prover-V2 (CoT) · FormalML retrieval subset | Pass@32 (%): 37.94 -> 47.71 | Improved |
| add 10 candidate premises to the prompt (ablation) | STP · FormalML retrieval subset | Pass@32 (%): 57.72 -> 56.43 | Hurt |
| add 10 candidate premises to the prompt (ablation) | Goedel-Prover · FormalML retrieval subset | Pass@32 (%): 39.04 -> 41.05 | Improved |
| add 10 candidate premises to the prompt (ablation) | DeepSeek-Prover-V1.5 · FormalML retrieval subset | Pass@32 (%): 49.45 -> 46.86 (Pass@16); 52.94 -> 53.59 (Pass@32) | Mixed |
| increase candidate premises from 10 to 20 (ablation) | provers in general · FormalML retrieval subset | Pass@32 (%): DeepSeek-Prover-V2 (noCoT) 69.29 -> 65.80 | Hurt |
| use long-CoT reasoning (DeepSeek-Prover-V2 CoT vs noCoT) (ablation) | DeepSeek-Prover-V2 · FormalML (All) | Pass@32 (%): 62.06 -> 39.50 | Hurt |
| increase sample budget from 1 to 32 (ablation) | Kimina-Prover-Preview-7B · FormalML (All) | Pass@K (%): 15.43 -> 45.74 | Improved |
| best-first tree search with larger budget (8x50 vs 4x20) (ablation) | BFS-Prover · FormalML (All) | Pass@K (%): 22.26 -> 25.31 | No clear effect |
| one round of expert iteration on mined subgoal problems | DeepSeek-Prover-V2 (noCoT) and Goedel-Prover-V2-8B · FormalML (All) | Pass@1: Pass@1 improved by about 20% | Improved |
| increase proof difficulty level (proof length 1 to 5) (ablation) | all whole-proof provers · FormalML | Pass@K | Hurt |
Closest papers by meaning
- Subgoal-based Demonstration Learning for Formal Theorem Proving (ICML 2024 · similarity 0.90)
- APOLLO: Automated LLM and Lean Collaboration for Advanced Formal Reasoning (NeurIPS 2025 · similarity 0.90)
- FaithSieve: Fine-Grained Evaluation of Math Proofs with Faithful Formal Evidence (NeurIPS 2026 · similarity 0.89)
- Not All Proofs Are Equal: Evaluating LLM Proof Quality Beyond Correctness (NeurIPS 2026 · similarity 0.89)
- Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs (ICML 2025 · similarity 0.88)
- ProofAug: Efficient Neural Theorem Proving via Fine-grained Proof Structure Analysis (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
