Research map / Large language models / Agents, code and math
FinMathBench: A Formula-Driven Benchmark for Evaluating LLMs’ Math Reasoning Capabilities in Finance
Yi He, Ping Wang, Shiqiang Xiong, Chao Chen, Haixiang Hu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase number of synthesized formulas from 1 to 4 | GPT-4o (CoT) · FinMathBench | accuracy: 72.9% -> 14.0% | Hurt |
| increase number of synthesized formulas from 1 to 4 | Gemini-2.5-Pro (CoT) · FinMathBench | accuracy: 88.4% -> 42.1% | Hurt |
| use Program-of-Thought instead of Chain-of-Thought | 40 LLMs · FinMathBench | accuracy: only 4 of 40 LLMs do better with CoT | Improved |
| use PoT (external execution) instead of CoT on 1-formula questions | average of 40 LLMs · FinMathBench 1-Formula | accuracy: CoT drops 20.2% vs PoT | Improved |
| use PoT instead of CoT on 4-formula questions | average of 40 LLMs · FinMathBench 4-Formula | accuracy: CoT drops 47.2% vs PoT | Improved |
| use PoT instead of CoT | o3-mini · FinMathBench (average) | accuracy: 58.4 -> 77.0 | Improved |
| use PoT instead of CoT | Gemini-2.5-Flash · FinMathBench (average) | accuracy: 41.8 -> 76.0 | Improved |
| use PoT instead of CoT | DeepSeek-Prover-V2 671B · FinMathBench (average) | accuracy: 65.9 (CoT) vs 62.9 (PoT) | Hurt |
| solve infrequently calculated target variable instead of frequently solved one | all 40 LLMs (CoT) · FinMathBench 1-Formula | accuracy: all 40 LLMs lower; 32 keep the gap under PoT | Hurt |
| prompt with extreme but valid fund fee rate 0.1089 | evaluated LLMs · FinMathBench 1-Formula (fee rate example) | error rate: 11 of 34 models modify it to 0.1089% | Hurt |
| prompt with extreme stock turnover rate 1.41 | evaluated LLMs · FinMathBench 1-Formula (turnover example) | error rate: 14 of 36 models convert it to 1.41% | Hurt |
| use math-specific model instead of general model of same size | Qwen2.5-Math-72B vs Qwen2.5-72B (CoT) · FinMathBench | accuracy: 41.9% -> 5.4% | Hurt |
Closest papers by meaning
- FinBen: A Holistic Financial Benchmark for Large Language Models (NeurIPS 2024 · similarity 0.90)
- FinSearchComp: Towards a Realistic, Expert-Level Evaluation of Financial Search and Reasoning (ICLR 2026 · similarity 0.88)
- STEER-ME: Assessing the Microeconomic Reasoning of Large Language Models (NeurIPS 2025 · similarity 0.87)
- FinMMDocR: Benchmarking Financial Multimodal Reasoning with Scenario Awareness, Document Understanding, and Multi-Step Computation (AAAI 2026 · similarity 0.87)
- EnterpriseBench: Evaluating LLM Agents on End-to-End Spreadsheet Tasks in Finance (NeurIPS 2026 · similarity 0.87)
- Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
