Research map / Large language models / Language, safety and interpretability

Controlling Equational Reasoning in Large Language Models with Prompt Interventions

AAAI 2025 · Large language models › Language, safety and interpretability · approach: Faithfulness of LLM explanations · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Uses a symbolic engine to generate fine-grained equational derivation data and applies prompt interventions (variable renaming, expression exchange, alternative goal, step removal) to study how LLM derivation errors change, comparing fine-tuned T5 against few-shot GPT and LLaMa-based models.

Problem

LLMs hallucinate and are sensitive to input perturbations when deriving mathematical equations; it is unclear how prompt interventions relate to specific types of derivation errors and whether reference-based metrics capture these errors.

Approach

Adapts and speeds up a symbolic data generation framework to produce fine-grained derivation data, applies targeted prompt interventions that map to distinct error-type footprints, and contrasts reference-based metrics with template-based error counts and human evaluation.

Key results

• FLAN-T5-large achieves the top scores on in-distribution generation metrics, outperforming few-shot GPT-4 across ROUGE, BLEU, BLEURT and GLEU.
• Human evaluation gives GPT-4 and FLAN-T5-Large 98% in-distribution equational reasoning accuracy, while Llemma-7B scores 70%.
• Step Removal is the hardest intervention in human evaluation (GPT-4 80%, FLAN-T5-Large 76%, Llemma-7B 36%), and fine-tuned models' out-of-distribution performance is overstated by generation metrics.

Measured effects reported in the paper

Surprising findings

• Reference-based generation metrics strongly disagree with human and template-based error evaluations on model rankings and perturbation difficulty.
• Fine-tuned T5 models that outperform GPT-4 on generation metrics do not outperform GPT-4 or GPT-3.5 on out-of-distribution examples by error count.

Limitations

• Human evaluation covers only 750 derivations sampled from a small set of models.
• Reference-based metrics are misaligned with error-based and human judgments, limiting their use for evaluating interventions.
• Few-shot GPT and LLaMa results rely on 5-shot prompting and reference-based sampling is limited to 100-2K examples.

Experimental setup

Datasets: symbolic-engine generated derivation dataset

Benchmarks: in-distribution static test set Variable Renaming Expression Exchange Alternative Goal Step Removal

Metrics: ROUGE BLEU BLEURT GLEU template-based error counts human accuracy

Baselines: GPT-4 GPT-3.5 LLaMa-2-7B Llemma-7B

Builds on: symbolic derivation data generation framework (prior work)

Closest papers by meaning

• Neuro-Symbolic Data Generation for Math Reasoning (NeurIPS 2024 · similarity 0.90)
• Improving Factuality and Reasoning in Language Models through Multiagent Debate (ICML 2024 · similarity 0.89)
• A Careful Examination of Large Language Model Performance on Grade School Arithmetic (NeurIPS 2024 · similarity 0.89)
• Frugal LMs Trained to Invoke Symbolic Solvers Achieve Parameter-Efficient Arithmetic Reasoning (AAAI 2024 · similarity 0.89)
• Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems (ICLR 2025 · similarity 0.89)
• Interpreting and Improving Large Language Models in Arithmetic Calculation (ICML 2024 · similarity 0.89)