Research map / Large language models / Agents, code and math
Exploring Equation as a Better Intermediate Meaning Representation for Numerical Reasoning of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BRIDGE, a prompting pipeline (erase, decompose, translate, answer) that makes LLMs generate equations instead of programs as intermediate meaning representations for numerical reasoning, with a theoretical proposition that equations are at least as accurate as programs.
Problem
LLMs generate equations as intermediate meaning representations for math word problems worse than programs, even though equations are semantically closer to questions; the paper asks how to theoretically justify and empirically improve equation generation.
Approach
Erase the asking part of the question and decompose the remaining text into sub-questions so that few-shot LLMs translate into equations rather than constant expressions or programs; add a regenerate-with-rising-temperature loop until the final equation system is solvable, and prove via a subset/hop-number proposition that equation IMRs should have higher generation accuracy than programs.
Key results
- BRIDGE with Codex reaches 74.2 on GSM8K, 86.1 on SVAMP and 78.5 on Algebra, above PoT/PAL under single-path setting.
- With gpt-3.5-turbo, BRIDGE scores 77.2 on GSM8K, 82.3 on SVAMP and 82.0 on Algebra (PoT: 74.8, 79.3, 64.0).
- Removing the Erase stage lowers Codex accuracy on GSM8K from 74.2 to 69.0 and on Algebra from 78.5 to 72.1.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use BRIDGE instead of the previous SOTA method | Codex · GSM8K | exact match: 72.0 -> 74.2 | Helped |
| use BRIDGE instead of the previous SOTA method | Codex · SVAMP | exact match: 85.2 -> 86.1 | Helped |
| use BRIDGE instead of the previous SOTA method | Codex · Algebra | exact match: 76.3 -> 78.5 | Helped |
| add the Erase stage (ablation) | BRIDGE with Codex · GSM8K | exact match: 69.0 -> 74.2 | Helped |
| add the Decompose stage (ablation) | BRIDGE with Codex · GSM8K | exact match: 68.2 -> 74.2 | Helped |
| add the Erase stage (ablation) | BRIDGE with Codex · SVAMP | exact match: 86.0 -> 86.1 | No clear effect |
| add the Decompose stage (ablation) | BRIDGE with Codex · SVAMP | exact match: 85.6 -> 86.1 | Helped |
| add the Erase stage (ablation) | BRIDGE with Codex · Algebra | exact match: 72.1 -> 78.5 | Helped |
| add the Decompose stage (ablation) | BRIDGE with Codex · Algebra | exact match: 77.8 -> 78.5 | Helped |
| add the Erase stage (ablation) | BRIDGE with GPT3.5 · GSM8K | exact match: 76.1 -> 77.0 | Helped |
| add the Decompose stage (ablation) | BRIDGE with GPT3.5 · GSM8K | exact match: 74.2 -> 77.0 | Helped |
| add the Erase stage (ablation) | BRIDGE with GPT3.5 · SVAMP | exact match: 81.9 -> 82.5 | Helped |
Surprising findings
- Equations are reported to generate worse than programs for current LLMs by default, attributed to pre-training data scarcity of equations.
- Gains on SVAMP are small because most failures there require language understanding rather than changing IMR.
Limitations
- Evaluated only on two closed OpenAI models (code-davinci-002 and gpt-3.5-turbo) and three datasets.
- Improvements are small and sometimes within run-to-run variance, e.g. the Algebra fluctuation exceeds 1.7%.
- Error analysis shows translation and answer stages remain weak, with LLMs still confusing question semantics and equations.
Experimental setup
Datasets: GSM8K SVAMP Algebra
Benchmarks: GSM8K SVAMP Algebra
Metrics: exact match proportion of equation-based correct answers equation counts with and without Erase
Baselines: CoT Tab-CoT Declarative PoT PAL
Builds on: program-of-thoughts PAL chain-of-thought self-consistency
Closest papers by meaning
- Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages (ICLR 2025 · similarity 0.88)
- MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models (ICLR 2024 · similarity 0.88)
- MathCoder2: Better Math Reasoning from Continued Pretraining on Model-translated Mathematical Code (ICLR 2025 · similarity 0.88)
- GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (ICLR 2025 · similarity 0.88)
- MathFimer: Enhancing Mathematical Reasoning by Expanding Reasoning Steps through Fill-in-the-Middle Task (ICLR 2026 · similarity 0.88)
- Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.