Research map / Large language models / Agents, code and math
Local Success Does Not Compose: Benchmarking Large Language Models for Compositional Formal Verification
Xu Xu, Xin Li, Xingwei Qu, Jie Fu, Binhang Yuan
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| sample more attempts, Pass@1 to Pass@8 | all 13 LLMs · DafnyComp | verified rate: average gain from Pass@4 to Pass@8 only 0.8% | No clear effect |
| sample more attempts, Pass@1 to Pass@8 | all 13 LLMs · DafnyComp | syntax correct rate: +7.3% on average (Pass@4 to Pass@8) | Improved |
| sample more attempts, Pass@1 to Pass@8 | claude-3.5-sonnet · DafnyComp | verified rate: 3.67 -> 7.00 | Improved |
| use reasoning-specialized models (QwQ-32B, DeepSeek-R1) | LLMs · DafnyComp | verified rate: QwQ-32B 0.00 at Pass@8; DeepSeek-R1 0.33 | No clear effect |
| scale model size (480B vs 32B) | Qwen coder models · DafnyComp | verified rate: 1.00 vs 0.67 at Pass@8 | No clear effect |
| move from single-function (DafnyBench) to multi-function (DafnyComp) tasks | LLMs · Dafny specification generation | verification success: ~53% -> 3.69% | Hurt |
| use code-specialized instead of general-purpose models | LLMs · DafnyComp | verified rate | No clear effect |
| use chain-based composition instead of arbitrary call graphs (ablation) | benchmark synthesis · synthesis success | success rate: <8% -> 47% | Improved |
| use incremental AST-guided translation with verifier refinement instead of direct Python-to-Dafny translation (ablation) | benchmark synthesis · Python to Dafny translation | success rate: <5% -> 47% | Improved |
Closest papers by meaning
- Evaluating Program Semantics Reasoning with Type Inference in System $F$ (NeurIPS 2025 · similarity 0.88)
- Variation in Verification: Understanding Verification Dynamics in Large Language Models (ICLR 2026 · similarity 0.88)
- Lemur: Integrating Large Language Models in Automated Program Verification (ICLR 2024 · similarity 0.88)
- CoRe: Benchmarking LLMs’ Code Reasoning Capabilities through Static Analysis Tasks (NeurIPS 2025 · similarity 0.88)
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025 · similarity 0.87)
- BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
