Research map / Large language models / Agents, code and math
One Example Shown, Many Concepts Known! Counterexample-Driven Conceptual Reasoning in Mathematical LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CounterMath, a 1,216-item university-level counterexample-based math judgement benchmark, evaluates many LLMs on it, and fine-tunes Qwen2.5-Math-7B-Instruct with 1,025 filtered SFT samples.
Problem
Current mathematical LLMs prove statements mainly when they have seen similar proof processes in training; they lack conceptual understanding that counterexample-driven proof would reveal.
Approach
Evaluate and train LLMs to judge true/false of university-level statements and justify with counterexamples (proof by counterexample), using a benchmark built from counterexample textbooks plus an LLM-based filter/refine pipeline for SFT data from existing proof datasets.
Key results
- Gemini2.5-pro reaches 77.0 F1 and Deepseek-R1 reaches 80.7 F1 on CounterMath, while open-source models mostly score around 30 F1.
- Qwen2.5-Math-7B-Instruct with SFT on 1,025 samples improves F1 from 38.3 to 39.7 (41.1 with hint prompt).
- The SFT model improves MATH from 80.5 to 87.9 and AIME 2024 from 20.0 to 30.0 over the base 7B model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Qwen2.5-Math-7B-Instruct with CounterMath-oriented SFT data | Qwen2.5-Math-7B-Instruct · CounterMath | F1 (macro): 38.3 -> 39.7 | Helped |
| Add a prompt encouraging reasoning by examples (ablation) | Qwen2.5-Math-7B-Instruct · CounterMath | F1 (macro): 38.3 -> 39.4 | Helped |
| Add a prompt encouraging reasoning by examples (ablation) | Qwen2.5-Math-7B-Instruct · CounterMath | Strict (%): 30.2 -> 33.1 | Helped |
| Add a hint prompt to the SFT model (ablation) | Qwen2.5-Math-7B-Instruct-SFT · CounterMath | Strict (%): 31.4 -> 31.1 | Hurt |
| Fine-tune Qwen2.5-math-7B-Instruct with CounterMath-oriented SFT data | Qwen2.5-math-7B-Instruct · GSM8K | score: 95.1 -> 95.6 | Helped |
| Fine-tune Qwen2.5-math-7B-Instruct with CounterMath-oriented SFT data | Qwen2.5-math-7B-Instruct · MATH | score: 80.5 -> 87.9 | Helped |
| Fine-tune Qwen2.5-math-7B-Instruct with CounterMath-oriented SFT data | Qwen2.5-math-7B-Instruct · Olympaidbench-Math | score: 41.6 -> 46.4 | Helped |
| Fine-tune Qwen2.5-math-7B-Instruct with CounterMath-oriented SFT data | Qwen2.5-math-7B-Instruct · MMLU-college-Math | score: 74.0 -> 80.0 | Helped |
| Fine-tune Qwen2.5-math-7B-Instruct with CounterMath-oriented SFT data | Qwen2.5-math-7B-Instruct · AIME 2024 | score: 20.0 -> 30.0 | Helped |
Surprising findings
- Some 7B models (e.g., Deepseek-Math-7B-RL) outperform the 70B WizardMath-70B-v1.0 on the benchmark.
- In-context learning prompts did not significantly change LLM performance on the counterexample benchmark.
Limitations
- Fine-tuning uses only a simple SFT strategy on a small dataset, and the trained model is slightly worse on some metrics.
- Textbook-derived statements are mostly True; about 5% were manually modified to create False labels.
- Evaluation relies on lexical F1 and a GPT-4o judge for example alignment.
Experimental setup
Datasets: ProofNet NaturalProof CounterMath
Benchmarks: CounterMath GSM8K MATH OlympiadBench MMLU-college-Math AIME 2024
Metrics: F1 (macro) Examples (%) Strict align (%) Loose align (%)
Baselines: Deepseek-Math-7B-RL Eurus-2-7B-PRIME Qwen2.5-Math-7B-Instruct Qwen2.5-Math-72B-Instruct GPT-4o OpenAI o1-preview Deepseek-R1 Claude3.7-sonnet Gemini2.5-pro QwQ-32B-Preview
Builds on: chain-of-thought Qwen2.5-Math AutoRace
Closest papers by meaning
- Not All Proofs Are Equal: Evaluating LLM Proof Quality Beyond Correctness (NeurIPS 2026 · similarity 0.91)
- MathArena: Evaluating LLMs on Uncontaminated Math Competitions (NeurIPS 2025 · similarity 0.91)
- RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics (NeurIPS 2025 · similarity 0.90)
- Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs (ICML 2025 · similarity 0.90)
- UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models (ICLR 2025 · similarity 0.90)
- CogMath: Assessing LLMs' Authentic Mathematical Ability from a Human Cognitive Perspective (ICML 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.