Research map / Large language models / Agents, code and math
ASyMOB: Algebraic Symbolic Mathematical Operations Benchmark
Michael Shalyt, Rotem Elimelech, Ido Kaminer
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add perturbations to seed questions | LLMs on symbolic math · ASyMOB | average success rate: 74.6% -> 46.8% | Hurt |
| enable code execution (ablation) | GPT-4o · ASyMOB | success rate: up to 37.7% in a single category | Improved |
| enable code execution (ablation) | Gemini-2.0 Flash · ASyMOB | success rate: up to 8.6% in a single category | Improved |
| enable code execution (ablation) | frontier models (o4-mini, Gemini 2.5 Flash) · ASyMOB | total success rate: o4-mini 78.1 -> 76.1; Gemini-2.5 Flash 78.5 -> 69.5 | No clear effect |
| enable code execution (ablation) | GPT-4.1 · ASyMOB | total success rate: 48.9 -> 46.2 total; seed 79 -> 83 | Mixed |
| replace symbols with numbers (Numeric-All-0, all ones) (ablation) | LLMs · ASyMOB | success rate | Hurt |
| use symbolic instead of one-numeric perturbation (ablation) | LLMs · ASyMOB | success rate | Hurt |
| add equivalence-preserving rewrites of constants (ablation) | most LLMs · ASyMOB | success rate | Hurt |
| use several Easy equivalence perturbations instead of one Hard (ablation) | most LLMs · ASyMOB | success rate | Hurt |
| add the seed question and answer as in-context exemplar (ablation) | Gemini 2.0 Flash, GPT-4o, Nemotron Super · ASyMOB Numeric-All-0 | success rate: +2%, +27%, +43.5% | Improved |
| add the seed question and answer as in-context exemplar (ablation) | Gemini 2.0 Flash, GPT-4o, Nemotron Super · ASyMOB Numeric-One-3, Numeric-All-3, Symbolic-3 | success rate: +2%, +5.1%, +6.8% | No clear effect |
| fine-tune on proofs (DeepSeek-Prover-V2 vs DeepSeek-V3 base) | DeepSeek-V3 · ASyMOB | total success rate: 45.4 -> 46.3 | No clear effect |
Closest papers by meaning
- Llemma: An Open Language Model for Mathematics (ICLR 2024 · similarity 0.87)
- DAG-Math: Graph-Guided Mathematical Reasoning in LLMs (ICLR 2026 · similarity 0.87)
- UGMathBench: A Diverse and Dynamic Benchmark for Undergraduate-Level Mathematical Reasoning with Large Language Models (ICLR 2025 · similarity 0.86)
- HARDMath2: A Benchmark for Applied Mathematics Built by Students as Part of a Graduate Class (NeurIPS 2025 · similarity 0.86)
- Frugal LMs Trained to Invoke Symbolic Solvers Achieve Parameter-Efficient Arithmetic Reasoning (AAAI 2024 · similarity 0.86)
- GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
