Research map / Large language models / Agents, code and math
REFACTOR: Learning to Extract Theorems from Proofs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains a GraphSAGE-based node classifier on Metamath proof trees, built by expanding human proofs with known theorems, to predict which subtree forms a reusable theorem, then uses it to extract, verify, and refactor new theorems and improve Holophrasm proving.
Problem
Automatically discovering modular, reusable theorems (lemmas) from formal mathematical proofs, which humans do by recognizing reusable subcomponents, in the Metamath formal system.
Approach
Reverse human theorem extraction: expand a proof by inlining a known theorem's proof, creating a synthetic proof tree where the inlined subtree is a labeled target; train a GNN to do node-level binary classification of which nodes form the theorem, then verify predicted subtrees as valid theorems.
Key results
- REFACTOR achieves 84.3% test node accuracy and 13.3% test proof accuracy with the 1.2M-parameter model, rising to 19.6% proof accuracy with the 4.5M-parameter model.
- The symbolic compression baseline reaches only 1.7% of extracted theorems matching set.mm theorems, versus 19.6% for REFACTOR.
- Using REFACTOR-augmented data, Holophrasm'20 proves 632 of 2720 test theorems, versus 557 for Holophrasm'20 and 600 for MetaGen.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use REFACTOR instead of the symbolic compression baseline for theorem extraction | Theorem extraction model · Theorem extraction test set | proof accuracy: 1.7% -> 19.6% | Helped |
| Train Holophrasm on the refactored dataset with newly extracted theorems | Holophrasm'20 · Metamath test theorems | test proofs found: 557 -> 632 | Helped |
| Train Holophrasm on the refactored dataset with newly extracted theorems | Holophrasm'16 · Metamath test theorems | test proofs found: 412 -> 468 | Helped |
| Remove all graph edges while retaining node features (ablation) | REFACTOR · Theorem extraction test set | proof accuracy: 13.3% -> 0.1% | Hurt |
| Keep only edges from leaves to root (ablation) | REFACTOR · Theorem extraction test set | proof accuracy: 13.3% -> 0.1% | Hurt |
| Keep only edges from root to leaves (ablation) | REFACTOR · Theorem extraction test set | proof accuracy: 13.3% -> 3.5% | Hurt |
| Remove node features (ablation) | REFACTOR · Theorem extraction test set | proof accuracy: 13.3% -> 0% | Hurt |
| Increase model size from 80k to 4535k trainable parameters (ablation) | REFACTOR · Theorem extraction test set | proof accuracy: 2.3% -> 19.6% | Helped |
Surprising findings
- Edge direction matters: unidirectional leaves-to-root edges make identical hypothesis leaves indistinguishable, making proof-level prediction theoretically impossible.
- Theorems extracted from the human set.mm library were used on average 733.5 times, far more than expanded-dataset theorems (77.4 average).
Limitations
- Node-level independent predictions produce many disconnected components (non-tree predictions), so many predictions cannot form theorems.
- Proof-level accuracy is low (19.6%), and most predicted components are invalid.
- Expansion is restricted to one theorem at a time and simple schemes; more sophisticated expansion left for future work.
Experimental setup
Datasets: set.mm (Metamath library)
Benchmarks: set.mm theorem extraction test set Metamath test theorems (2720)
Metrics: node accuracy proof accuracy number of test proofs found new theorem usage
Baselines: Holophrasm'16 Holophrasm'20 MetaGen Symbolic compression baseline
Builds on: GraphSAGE Holophrasm Adam
Closest papers by meaning
- miniCTX: Neural Theorem Proving with (Long-)Contexts (ICLR 2025 · similarity 0.89)
- LEGO-Prover: Neural Theorem Proving with Growing Libraries (ICLR 2024 · similarity 0.88)
- ProofAug: Efficient Neural Theorem Proving via Fine-grained Proof Structure Analysis (ICML 2025 · similarity 0.88)
- Alchemy: Amplifying Theorem-Proving Capability Through Symbolic Mutation (ICLR 2025 · similarity 0.87)
- Reviving DSP for Advanced Theorem Proving in the Era of Reasoning Models (NeurIPS 2025 · similarity 0.87)
- Subgoal-based Demonstration Learning for Formal Theorem Proving (ICML 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.