Research map / Large language models / Agents, code and math
Metacognitive Capabilities of LLMs: An Exploration in Mathematical Problem Solving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that LLMs can name their own math skills (via GPT-4 labeling and clustering of training questions) and that presenting skill-matched exemplars improves in-context math solving on GSM8K and MATH.
Problem
Whether LLMs possess metacognitive knowledge, specifically the ability to name the skills needed for a task, and whether using such self-identified skills can improve math problem solving via in-context examples.
Approach
A powerful LLM (GPT-4-0613) labels each training question with a fine-grained skill, clusters these into coarser skills, and builds a skill exemplar repository; at test time the model selects a skill from the list and receives randomly chosen exemplars for that skill as few-shot examples.
Key results
- On MATH with GPT-4-0613, CoT + Skill-Based reaches 53.88% overall versus 42.2% for CoT (about 11.6 points gain).
- PAL + Skill-Based (7 skill-based + 1 code example) reaches 62.00% on a 500-question MATH subset versus 52.0% for PAL (4-shot).
- On GSM8K with GPT-4-0613, CoT + Skill-Based gets 94.31% vs 93.00% CoT; with maj@5 it reaches 95.38%. Skill exemplars from GPT-4 also raise Mixtral-8x7B MATH accuracy from 28.4 (CoT maj@4) to 32.44 with maj@4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace standard CoT examples with skill-matched exemplars | GPT-4-0613 · GSM8K | accuracy: 93.00 -> 94.31 | Helped |
| Use random repository examples instead of skill-matched exemplars (ablation) | GPT-4-0613 CoT · GSM8K | accuracy: 93.00 -> 92.87 | Hurt |
| Add self-consistency with maj@5 to skill-based CoT (ablation) | GPT-4-0613 CoT + Skill-Based · GSM8K | accuracy: 94.31 -> 95.38 | Helped |
| Replace broad topic-selected examples with skill-matched exemplars | GPT-4-0613 CoT · MATH | accuracy: 50.31 -> 53.88 | Helped |
| Replace standard CoT examples with skill-matched exemplars | GPT-4-0613 · MATH | accuracy: 42.2 -> 53.88 | Helped |
| Use skill-based text exemplars with PAL | GPT-4-0613 PAL · MATH | accuracy: 52.0 -> 59.32 | Helped |
| Increase skill exemplars from 3 to 7 while retaining one code exemplar (ablation) | GPT-4-0613 PAL + Skill-Based · MATH | accuracy: 59.32 -> 62.00 | Helped |
| Use GPT-4 skill annotations instead of GPT-3.5 annotations (ablation) | GPT-4-0613 CoT + Skill-Based · MATH | accuracy: 51.9 -> 53.8 | Helped |
| Transfer GPT-4-labeled skill exemplars to Mixtral | Mixtral 8 × 7B CoT · MATH | accuracy: 28.4 -> 28.44 | Helped |
| Replace topic-based exemplars with skill-based exemplars | Mixtral 8 × 7B CoT · MATH | accuracy: 26.14 -> 28.44 | Helped |
| Add self-consistency (maj@4) to skill-based CoT (ablation) | Mixtral 8 × 7B CoT + Skill-Based · MATH | accuracy: 28.44 -> 32.44 | Helped |
| Transfer GSM8K skill exemplars to other math word-problem datasets | GPT-4-0613 CoT · SVAMP | accuracy: 91.9 -> 92.6 | Helped |
Surprising findings
- Llama-2 70B could not produce sensible skill names, and Mixtral-8x7B clustering collapsed all skills into one cluster, so only GPT-4 and GPT-3.5 were usable as skill annotators.
- Skill-based exemplar selection outperforms complexity-based selection (Complex CoT) on MATH, suggesting relevance matters more than reasoning-chain length.
Limitations
- Each question is assigned only one skill, ignoring secondary skills needed for multi-step problems.
- Relies on GPT-4 for skill labeling; method only demonstrated for in-context learning, not fine-tuning.
- Only one run per setting in many experiments with GPT-4 at temperature 1.0; improvements are modest on GSM8K (about 1.3 points).
Experimental setup
Datasets: GSM8K MATH SVAMP ASDIV MAWPS (SingleOP, SingleEQ, AddSub, MultiArith)
Benchmarks: GSM8K MATH SVAMP ASDIV SingleOP SingleEQ AddSub MultiArith
Metrics: accuracy
Baselines: CoT Complex CoT Topic-Based Random Retrieval-RSD PAL CoT + PAL
Builds on: chain-of-thought prompting program-aided language models ComplexCoT self-consistency
Closest papers by meaning
- CogMath: Assessing LLMs' Authentic Mathematical Ability from a Human Cognitive Perspective (ICML 2025 · similarity 0.90)
- Advancing Mathematical Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages (ICLR 2025 · similarity 0.88)
- Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process (ICLR 2025 · similarity 0.88)
- Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models (ICLR 2025 · similarity 0.88)
- Assessing the Creativity of LLMs in Proposing Novel Solutions to Mathematical Problems (AAAI 2025 · similarity 0.87)
- OMEGA: Can LLMs Reason Outside the Box in Math? Evaluating Exploratory, Compositional, and Transformative Generalization (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.