Research map / Large language models / Agents, code and math
PAL: Program-aided Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Generate program steps and execute them with Python instead of relying on the LM to calculate | Codex few-shot reasoning · GSM8K | solve rate (%): 65.6 -> 72.0 | Helped |
| Generate program steps and execute them with Python instead of relying on the LM to calculate | Codex few-shot reasoning · GSM-hard | solve rate (%): 23.1 -> 61.2 | Helped |
| Use program-aided reasoning instead of CoT | Codex few-shot reasoning · SVAMP | solve rate (%): 74.8 -> 79.4 | Helped |
| Use program-aided reasoning instead of CoT | Codex few-shot reasoning · Colored Objects | solve rate: 86.3 -> 95.1 | Helped |
| Use program-aided reasoning instead of CoT | Codex few-shot reasoning · Penguins | solve rate: 79.2 -> 93.3 | Helped |
| Use program-aided reasoning instead of CoT | Codex few-shot reasoning · Date | solve rate: 64.8 -> 76.2 | Helped |
| Use program-aided reasoning instead of CoT | Codex few-shot reasoning · Repeat Copy | solve rate: 68.8 -> 90.6 | Helped |
| Use program-aided reasoning instead of CoT | Codex few-shot reasoning · Object Counting | solve rate: 73.0 -> 96.7 | Helped |
| Use nucleus sampling and majority voting over 40 samples | PaL on Codex · GSM8K | solve rate (%): 72.0 -> 80.4 | Helped |
| Use a single-line expression instead of multi-step code (ablation) | PaL · GSM8K | solve rate (%): 72.0 -> 47.8 | Hurt |
| Have the LLM produce the answer from its code rather than execute it with Python (ablation) | PaL · GSM8K | solve rate (%): 72.0 -> 23.2 | Hurt |
| Use meaningful variable names instead of single-letter names (ablation) | PaL · GSM8K | solve rate (%): 59 -> 71.8 | Helped |
Closest papers by meaning
- Large Language Models as Tool Makers (ICLR 2024 · similarity 0.88)
- Reasoning-as-Logic-Units: Scaling Test-Time Reasoning in Large Language Models Through Logic Unit Alignment (ICML 2025 · similarity 0.88)
- LLM Augmented LLMs: Expanding Capabilities through Composition (ICLR 2024 · similarity 0.88)
- Frugal LMs Trained to Invoke Symbolic Solvers Achieve Parameter-Efficient Arithmetic Reasoning (AAAI 2024 · similarity 0.87)
- Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning (ICML 2025 · similarity 0.87)
- Exploring Equation as a Better Intermediate Meaning Representation for Numerical Reasoning of Large Language Models (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.