Research map / Large language models / Agents, code and math
How Powerful are LLMs in Generating Formal Program Specifications?
Fanpeng Yang, Xing Li, Shuling Wang, Jie An, Zeyu Sun, Shenghua Feng, Wenhan Wang, Weiyi Wang, Naijun Zhan, Fanjiang Xu
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| verify with stronger Gemini 3 Pro Preview instead of self-verification (ablation) | COINS pipeline · COINS HumanEval | PASSall: +3.05% average | Improved |
| use human-written specs instead of the model's own specs (ablation) | COINS pipeline · COINS HumanEval | PASSall: +5.01% average | Improved |
| allow Fixpoint definitions in human specifications (ablation) | human-written Rocq specs · HumanEval, Gemini 3 Pro Preview prover | proof success rate | Improved |
| use COINS instead of CLEVER-style equivalence proofs | specification evaluation · HumanEval | discriminative power across models: 0.621% (CLEVER) vs 1.22% to 28.05% REJECTall | Improved |
| require all positive tests rather than only the first (ablation) | Gemini 3 Pro Preview specs · COINS HumanEval | pass rate: 60.98% -> 28.05% | Hurt |
| add negative-case rejection (REJECTall) after PASSall (ablation) | COINS pipeline · COINS HumanEval | candidate specs: PASSall 28.05% vs REJECTall 28.05% for Gemini 3 Pro Preview | No clear effect |
| use a newer frontier model for spec generation | spec generation · COINS HumanEval | syntax validity: 7.93% (DeepSeek-V3.1) -> 78.05% (Gemini 3 Pro Preview) | Improved |
Closest papers by meaning
- APOLLO: Automated LLM and Lean Collaboration for Advanced Formal Reasoning (NeurIPS 2025 · similarity 0.87)
- FVEL: Interactive Formal Verification Environment with Large Language Models via Theorem Proving (NeurIPS 2024 · similarity 0.86)
- How efficient is LLM-generated code? A rigorous & high-standard benchmark (ICLR 2025 · similarity 0.86)
- Synthetic Programming Elicitation for Text-to-Code in Very Low-Resource Programming and Formal Languages (NeurIPS 2024 · similarity 0.86)
- Beyond Accuracy: Evaluating Self-Consistency of Code Large Language Models with IdentityChain (ICLR 2024 · similarity 0.86)
- Lemur: Integrating Large Language Models in Automated Program Verification (ICLR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
