Research map / Large language models / Agents, code and math

PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts

NeurIPS 2025 · Large language models › Agents, code and math · approach: Reasoning benchmarks and platforms

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces PolyMath, a 4-level, 18-language mathematical reasoning benchmark (9000 parallel problems) with human-calibrated translations and a difficulty-weighted accuracy metric, and evaluates 16 LLMs plus language-consistency and language-control analyses.

Problem

Existing multilingual math benchmarks (MGSM, MSVAMP) are too easy for reasoning LLMs, and advanced math benchmarks are mostly English-only, leaving the relationship between language and reasoning under-studied.

Approach

A multilingual math benchmark organized into four difficulty levels built from thought depth and knowledge breadth, with 18 parallel languages translated by GPT-4o pre-translation followed by expert human calibration, scored with a difficulty-weighted accuracy (weights doubling per level).

Key results

• Qwen-3-235B-A22B-Thinking reaches the highest DW-ACC benchmark score of 54.6 and Gemini-2.5-pro 52.2; about 40% accuracy at the top level.
• Reasoning models show language-consistency around 40% in thinking for Qwen-QwQ-32B and Deepseek-R1-671B while non-reasoning models stay near 100%.
• Forcing English reasoning gives the best overall DW-ACC for Qwen-QwQ-32B (47.9 vs 45.9 unconstrained), while forcing output in the query language lowers it to 43.4.

Measured effects reported in the paper

Surprising findings

• Forcing reasoning models to respond in the query language often degrades performance, while forcing English thinking improves it.
• Reasoning models can underperform non-reasoning models at the low (K-12) level, suggesting low-difficulty benchmarks fail to reflect reasoning ability.

Limitations

• Translation pipeline relies on GPT-4o pre-translation, and the authors report nonzero content-error rates in those drafts.
• Only 125 problems per level, so per-language sample size is small and results have sampling noise.
• Answer verification uses rule-based matching with over 98% precision on sampled inspection, not perfect.

Experimental setup

Datasets: MGSM P-MMeval HLE AMC AIME Gaokao IMO Putnam CMO USAMO

Benchmarks: PolyMath

Metrics: accuracy (average@16) difficulty-weighted accuracy (DW-ACC) standard deviation across languages language range input-output language consistency

Baselines: Qwen-3-235B-A22B-Thinking Gemini-2.5-pro Deepseek-R1-671B Qwen-QwQ-32B OpenAI-o3-mini-medium OpenAI-o1-mini Claude-3.7-sonnet-thinking Gemini-2.0-flash-thinking GPT-4.5-preview ChatGPT-4o-latest Deepseek-v3 Qwen-2.5-Max

Builds on: MGSM MSVAMP HLE

Closest papers by meaning

• Multilingual Mathematical Reasoning: Advancing Open-Source LLMs in Hindi and English (AAAI 2025 · similarity 0.91)
• RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics (NeurIPS 2025 · similarity 0.91)
• UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models (ICLR 2025 · similarity 0.91)
• Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs (ICML 2025 · similarity 0.90)
• GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (ICLR 2025 · similarity 0.90)
• CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery (ICLR 2025 · similarity 0.90)