Research map / Large language models / Agents, code and math

RMath: A Logic Reasoning-Focused Datasets Toward Mathematical Multistep Reasoning Tasks

AAAI 2025 · Large language models › Agents, code and math · approach: Reasoning benchmarks and platforms · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds RMath, a 200-problem logic-focused math reasoning dataset with propositional annotations, and a proposition-based problem-solving framework used for prompt tuning of LLMs evaluated on RMath.

Problem

Existing math datasets emphasize commonsense knowledge or fixed-format arithmetic rather than pure multistep logical reasoning from given conditions, so LLM logical reasoning ability is not well measured.

Approach

A dataset of 200 knowledge-free multistep logic problems (direct, hypothetical, nested) annotated with atomic propositions, truth values, and proposition relations, plus a nine-step propositional-logic solving framework used to create prompt-tuning training data.

Key results

• DeepSeekMath 7B scores 45.0% on RMath while ToRA 70B scores only 31.0% despite >80% on GSM8K, SVAMP and ASDIV.
• Prompt tuning on RMath-train raises RMath accuracy by about 5.3% on average across models, with Llama3 70B gaining the most (up to 31%, from 33.0% to 64.0%).
• Llama2 7B and 13B score 0% on RMath before tuning.

Measured effects reported in the paper

Surprising findings

• Some models (WizardMath 13B, MetaMath 70B) perform worse on RMath after prompt tuning than before.
• ToRA 70B, which exceeds 80% on several standard math benchmarks, reaches only 31% on RMath.

Limitations

• Dataset is small, with only 200 problems.
• Problems were filtered to exclude commonsense and very hard problems, and multi-answer problems were reduced to single answers.
• Prompt-tuning gains are inconsistent, with some models degrading after tuning, and the explanation is only speculative.

Experimental setup

Datasets: RMath GSM8K MATH SVAMP ASDIV MAWPS TabMWP

Benchmarks: RMath GSM8K MATH SVAMP ASDIV MAWPS TabMWP

Metrics: accuracy

Baselines: Llama2 Llama3 WizardMath MetaMath ToRA DeepSeekMath

Builds on: MATH GSM8K ToRA MetaMath WizardMath

Closest papers by meaning

• Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages (ICLR 2025 · similarity 0.91)
• MathFimer: Enhancing Mathematical Reasoning by Expanding Reasoning Steps through Fill-in-the-Middle Task (ICLR 2026 · similarity 0.90)
• UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models (ICLR 2025 · similarity 0.90)
• Premise-Augmented Reasoning Chains Improve Error Identification in Math reasoning with LLMs (ICML 2025 · similarity 0.89)
• Evaluating Mathematical Reasoning Beyond Accuracy (AAAI 2025 · similarity 0.89)
• OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalization (NeurIPS 2025 · similarity 0.89)