Research map / Large language models / Agents, code and math
Benchmarking LLMs’ Mathematical Reasoning with Unseen Random Variables Questions
Zijin Hong, Hao Wu, Su Dong, Junnan Dong, Yilin Xiao, Yujing Zhang, Zhu Wang, Feiran Huang, Linyi Li, Hongxia Yang, Xiao Huang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace fixed-variable original problems with random-variable questions | LLMs on MATH · MATH-SP -> MATH-RV | accuracy: GPT-4o 83.48 -> 76.70; Llama-3.1-8B-It 57.39 -> 46.43 | Hurt |
| replace original LeetCode-Math problems with random-variable questions | LLMs on LeetCode-Math · LeetCode-SP -> LeetCode-RV | accuracy: GPT-4o 61.74 -> 50.09 | Hurt |
| apply test-time scaling (pass@10 vs single attempt) (ablation) | Llama-3.2-3B-It · LeetCode-RV | accuracy: 26.67% -> 56.52% | Improved |
| use stronger models (higher MATH-RV accuracy) | LLMs · MATH-RV robustness | accuracy drop: Pearson r = -0.72 | Improved |
| higher LeetCode-RV accuracy vs robustness | LLMs · LeetCode-RV robustness | accuracy drop: Pearson r = -0.14 | No clear effect |
| use reasoning models (o3-mini, DeepSeek-R1) vs chat LLMs like GPT-4o | RV-Bench · RV-Bench overall | accuracy: 85.05 / 82.35 vs 63.40 | Improved |
Closest papers by meaning
- Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs (ICML 2025 · similarity 0.92)
- GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (ICLR 2025 · similarity 0.91)
- Evaluating LLMs When They Do Not Know the Answer: Statistical Evaluation of Mathematical Reasoning via Comparative Signals (ICML 2026 · similarity 0.90)
- Evaluating Mathematical Reasoning Beyond Accuracy (AAAI 2025 · similarity 0.90)
- UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models (ICLR 2025 · similarity 0.90)
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark (ICLR 2025 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
