Research map / Large language models / Agents, code and math
RE-IMAGINE: Symbolic Benchmark Synthesis for Reasoning Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces RE-IMAGINE, a three-level (observe/mutate/imagine) reasoning hierarchy and an automated pipeline that converts benchmark problems to symbolic code, mutates them, and back-translates them to natural language to generate unseen variations, showing LLM performance drops on GSM8K, CLadder, CRUXEval, and Loop.
Problem
Existing LLM reasoning benchmark accuracy may reflect memorization or leakage rather than genuine reasoning, and current variation methods are manual, ad hoc, and hard to scale across benchmarks.
Approach
Mutate each benchmark problem in an executable symbolic (code or computational-graph) form using a hierarchy of Level-2 (mutate) and Level-3 (imagine/counterfactual) mutations, then translate back to natural language with GPT-4o, so ground-truth answers are computed automatically and the pipeline scales across benchmarks.
Key results
- Most models reach about 95% accuracy on the original GSM8K test set with 8-shot CoT, but nearly all drop around 10% on SampleValues mutations and Level-3 mutations pose the greatest challenge.
- On CLadder, Level-3 CounterFactual mutations cause roughly a 20% accuracy drop relative to the original test set.
- On GSM8K binary counterfactuals, o1 is the best model on N-IR/S-IR, while Llama 8B models, GPT-3.5, and GPT-4 are at the weaker end.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply SampleValues by changing numeric values while preserving the reasoning path | LLMs evaluated on GSM8K · GSM8K | accuracy: around 10% | Hurt |
| Add irrelevant information with UselessInfo | LLMs evaluated on GSM8K · GSM8K | accuracy | No clear effect |
| Add Level-3 mutations that amend the original problem logic | LLMs evaluated on GSM8K · GSM8K | accuracy | Hurt |
| Add a CounterFactual mutation | Best-performing model in each family · CLadder | accuracy: approximately $20$% drop | Hurt |
| Add Level-2 and Level-3 mutations to code tasks | LLMs evaluated on CRUXEval · CRUXEval | accuracy | Hurt |
| Add irrelevant variables and operations using X-Original | LLMs evaluated on Loop · Loop | success rate | Hurt |
| Replace junk variable names with names resembling original variables (Junk Hint to Junk No-Hint) | Smaller models evaluated on Loop · Loop | success rate | Hurt |
| Control for the number of reasoning steps when comparing mutated and original questions (ablation) | LLMs evaluated on GSM8K · GSM8K | accuracy | Hurt |
| Use Level-3 mutations rather than Level-2 mutations (ablation) | LLMs evaluated on GSM8K · GSM8K | accuracy | Hurt |
| Provide both original and corresponding mutated in-context examples instead of only original or only mutated examples (ablation) | Most models evaluated on GSM8K variations · GSM8K mutated test sets | accuracy | Helped |
| Compose two mutations rather than applying a single mutation (ablation) | LLMs evaluated on GSM8K · GSM8K mutated test sets | accuracy | Hurt |
| Add both SampleValues and another mutation (ablation) | LLMs evaluated on GSM8K · GSM8K mutated test sets | accuracy gap relative to original | Hurt |
Surprising findings
- Accuracy on Level-3 mutations with three reasoning steps is lower than on original problems requiring seven steps, suggesting the mutations themselves rather than complexity drive the drop.
- Composing two mutations widens the gap between mutated and original test sets.
Limitations
- Mutation pipeline introduces noise; human review found 0-6.67% invalid QA pairs per mutation type, and upper-bound accuracies are estimated by counting invalid items as correct.
- Binary counterfactual PN/PS analysis is limited to a simplified binarized GSM8K and only 50 validation questions due to computational cost.
- Loop benchmark mutations are restricted to Level-2 because substantial value changes can invalidate loop invariants.
Experimental setup
Datasets: GSM8K CLadder CRUXEval Loop OpenMathInstruct validation set
Benchmarks: GSM8K mutated variations CLadder mutated variations CRUXEval mutated variations Loop mutated variations
Metrics: accuracy N-IR S-IR PN PS
Baselines: GSM-Symbolic GSM-Hard GSM-IC iGSM CounterFactual benchmark
Builds on: Pearl ladder of causation OpenMathInstruct CRUXEval CLadder Frama-C
Closest papers by meaning
- Unveiling the Magic of Code Reasoning through Hypothesis Decomposition and Amendment (ICLR 2025 · similarity 0.91)
- Benchmarking Abstract and Reasoning Abilities Through A Theoretical Perspective (ICML 2025 · similarity 0.91)
- Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs (ICML 2025 · similarity 0.90)
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity (NeurIPS 2025 · similarity 0.90)
- ReJump: A Tree-Jump Representation for Analyzing and Improving LLM Reasoning (ICML 2026 · similarity 0.89)
- CodeCrash: Exposing LLM Fragility to Misleading Natural Language in Code Reasoning (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.