AI Paper Atlas

Research map / Large language models / Agents, code and math

FinMathBench: A Formula-Driven Benchmark for Evaluating LLMs’ Math Reasoning Capabilities in Finance

Yi He, Ping Wang, Shiqiang Xiong, Chao Chen, Haixiang Hu

AAAI 2026 · Large language models › Agents, code and math · approach: Mathematical and structural reasoning benchmarks · 1 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
increase number of synthesized formulas from 1 to 4GPT-4o (CoT) · FinMathBenchaccuracy: 72.9% -> 14.0%Hurt
increase number of synthesized formulas from 1 to 4Gemini-2.5-Pro (CoT) · FinMathBenchaccuracy: 88.4% -> 42.1%Hurt
use Program-of-Thought instead of Chain-of-Thought40 LLMs · FinMathBenchaccuracy: only 4 of 40 LLMs do better with CoTImproved
use PoT (external execution) instead of CoT on 1-formula questionsaverage of 40 LLMs · FinMathBench 1-Formulaaccuracy: CoT drops 20.2% vs PoTImproved
use PoT instead of CoT on 4-formula questionsaverage of 40 LLMs · FinMathBench 4-Formulaaccuracy: CoT drops 47.2% vs PoTImproved
use PoT instead of CoTo3-mini · FinMathBench (average)accuracy: 58.4 -> 77.0Improved
use PoT instead of CoTGemini-2.5-Flash · FinMathBench (average)accuracy: 41.8 -> 76.0Improved
use PoT instead of CoTDeepSeek-Prover-V2 671B · FinMathBench (average)accuracy: 65.9 (CoT) vs 62.9 (PoT)Hurt
solve infrequently calculated target variable instead of frequently solved oneall 40 LLMs (CoT) · FinMathBench 1-Formulaaccuracy: all 40 LLMs lower; 32 keep the gap under PoTHurt
prompt with extreme but valid fund fee rate 0.1089evaluated LLMs · FinMathBench 1-Formula (fee rate example)error rate: 11 of 34 models modify it to 0.1089%Hurt
prompt with extreme stock turnover rate 1.41evaluated LLMs · FinMathBench 1-Formula (turnover example)error rate: 14 of 36 models convert it to 1.41%Hurt
use math-specific model instead of general model of same sizeQwen2.5-Math-72B vs Qwen2.5-72B (CoT) · FinMathBenchaccuracy: 41.9% -> 5.4%Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy