Research map / Large language models / Agents, code and math
CogMath: Assessing LLMs' Authentic Mathematical Ability from a Human Cognitive Perspective
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CogMath, a cognition-inspired evaluation framework that tests LLM math mastery across 3 reasoning stages and 9 dimensions using a GPT-4-driven Inquiry-Judge-Reference multi-agent system, finding 30-40% overestimation of seven LLMs' abilities.
Problem
Standard math benchmarks rely on coarse final-answer accuracy, which overestimates LLMs' authentic mathematical abilities and hides where reasoning breaks down.
Approach
Formalizes human mathematical reasoning into problem comprehension, problem solving, and solution summarization, and generates dimension-specific inquiries (paraphrase, disruption, missing/redundant conditions, analogy, numeric transformation, knowledge redefinition, intermediate steps, backward reasoning) via an Inquiry-Judge-Reference multi-agent pipeline; a model masters a problem only if it pa
Key results
- Seven LLMs' pass rates drop by 30%-40% under CogMath relative to vanilla accuracy; GPT-4 passes only 39.3% on MATH and 67.1% on GSM8K.
- Stage 2 (problem solving) is the lowest-pass stage for GPT-4 (53.2%) and DeepSeek-V2.5 (52.6%) on MATH.
- CoT and one-shot ICL do not improve MATH pass rates (e.g., GPT-4 0.393 to 0.380 with CoT and 0.368 with ICL).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate GPT-4 with CogMath instead of the vanilla benchmark | GPT-4 · MATH Avg | pass rate: 0.758 -> 0.393 | Hurt |
| Evaluate GPT-4 with CogMath instead of the vanilla benchmark | GPT-4 · GSM8K | pass rate: 0.954 -> 0.671 | Hurt |
| Evaluate DeepSeek-V2.5 with CogMath instead of the vanilla benchmark | DeepSeek-V2.5 · MATH Avg | pass rate: 0.747 -> 0.368 | Hurt |
| Evaluate DeepSeek-R1 with CogMath instead of the vanilla benchmark | DeepSeek-R1 · MATH Avg | pass rate: 0.982 -> 0.448 | Hurt |
| Add Chain-of-Thought prompting (ablation) | GPT-4 · MATH | pass rate: 0.393 -> 0.380 | Hurt |
| Add one-shot in-context learning (ablation) | GPT-4 · MATH | pass rate: 0.393 -> 0.368 | Hurt |
| Add Chain-of-Thought prompting (ablation) | GPT-3.5 · MATH | pass rate: 0.176 -> 0.169 | Hurt |
| Add one-shot in-context learning (ablation) | GPT-3.5 · MATH | pass rate: 0.176 -> 0.167 | Hurt |
| Add Chain-of-Thought prompting (ablation) | Gemini-1.5 · MATH | pass rate: 0.291 -> 0.242 | Hurt |
| Add one-shot in-context learning (ablation) | Gemini-1.5 · MATH | pass rate: 0.291 -> 0.250 | Hurt |
| Add Chain-of-Thought prompting (ablation) | GPT-4 · GSM8K | pass rate: 0.671 -> 0.680 | Helped |
| Add one-shot in-context learning (ablation) | GPT-3.5 · GSM8K | pass rate: 0.424 -> 0.466 | Helped |
Surprising findings
- Stronger models score lower on counterfactual dimensions 2 and 3 than weaker Llama models, suggesting an 'over-correction' toward solvable problems.
- CoT and ICL reduce pass rates on MATH despite being widely reported to help reasoning.
Limitations
- Relies on multi-agent LLM interactions, limiting scalability and raising computational cost.
- Agents implemented with GPT-4, so the pipeline depends on a closed model; Program-of-Thoughts and RAG not studied.
Experimental setup
Datasets: GSM8K MATH MExam
Benchmarks: MATH GSM8K MExam
Metrics: pass rate relative pass rate
Baselines: GPT-4 GPT-3.5-Turbo Gemini-1.5-Flash DeepSeek-V2.5 Llama3-8B Llama2-13B Mixtral-8x7B-Instruct
Builds on: chain-of-thought in-context learning
Closest papers by meaning
- Assessing the Creativity of LLMs in Proposing Novel Solutions to Mathematical Problems (AAAI 2025 · similarity 0.91)
- RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics (NeurIPS 2025 · similarity 0.91)
- Atomic Thinking of LLMs: Decoupling and Exploring Mathematical Reasoning Abilities (NeurIPS 2025 · similarity 0.91)
- A Matter of Interest: Understanding Interestingness Judgments of Math Problems in Humans and Language Models (NeurIPS 2026 · similarity 0.91)
- Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving (NeurIPS 2024 · similarity 0.90)
- Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.