Research map / Large language models / Agents, code and math
FormalRx: Rectify and eXamine Semantic Failures in Autoformalization
Haocheng Wang, Baiyu Huang, Yingjia Wan, Xiao Zhu, Xiaoyang Liu, Yinya Huang, Zhijiang Guo
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use taxonomy-guided error synthesis instead of rule-based perturbation (ablation) | FormalRx-8B · FormalRx-Test verdict | macro F1: 0.100 -> 0.881 | Improved |
| use progressive training instead of joint training (ablation) | FormalRx-8B · FormalRx-Test correction | accuracy: 0.729 -> 0.792 | Improved |
| use progressive training instead of joint training (ablation) | FormalRx-8B · FormalRx-Test categorization | macro F1: 0.709 -> 0.761 | Improved |
| use progressive training instead of joint training (ablation) | FormalRx-8B · FormalRx-Test verdict | macro F1: 0.881 -> 0.899 | Improved |
| use progressive training instead of joint training (ablation) | FormalRx-8B · FormalRx-Test localization | accuracy: 0.750 -> 0.778 | Improved |
| use sequential training instead of progressive replay (ablation) | FormalRx-8B · FormalRx-Test unified output | output format | Hurt |
| fine-tune Qwen3-8B instead of Qwen2.5-7B (ablation) | FormalRx · FormalRx-Test correction | accuracy: 0.584 -> 0.729 | Improved |
| use Qwen2.5-7B backbone instead of Qwen3-8B (ablation) | FormalRx · FormalRx-Test categorization | macro F1: 0.709 -> 0.741 | Improved |
| use R1-Distill-Qwen-7B backbone instead of Qwen3-8B (ablation) | FormalRx · FormalRx-Test correction | accuracy: 0.729 -> 0.761 | Improved |
| fine-tune on taxonomy-annotated diagnostic data vs Qwen3-8B base | Qwen3-8B · FormalRx-Test categorization | macro F1: 0.155 -> 0.709 | Improved |
| scale FormalRx from 4B to 8B | FormalRx · FormalRx-Test correction | accuracy: 0.654 -> 0.729 | Improved |
| scale FormalRx from 4B to 8B | FormalRx · FormalRx-Test categorization | macro F1: 0.508 -> 0.709 | Hurt |
Closest papers by meaning
- FormalAlign: Automated Alignment Evaluation for Autoformalization (ICLR 2025 · similarity 0.89)
- Autoformalize Mathematical Statements by Symbolic Equivalence and Semantic Consistency (NeurIPS 2024 · similarity 0.87)
- ATLAS: Autoformalizing Theorems through Lifting, Augmentation, and Synthesis of Data (NeurIPS 2025 · similarity 0.87)
- Autonomous Evaluation of LLMs for Truth Maintenance and Reasoning Tasks (ICLR 2025 · similarity 0.85)
- Controlling Equational Reasoning in Large Language Models with Prompt Interventions (AAAI 2025 · similarity 0.85)
- Multi-language Diversity Benefits Autoformalization (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
