Research map / Large language models / Agents, code and math
StrategyLLM: Large Language Models as Strategy Generators, Executors, Optimizers, and Evaluators for Problem Solving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes StrategyLLM, a four-agent framework (strategy generator, executor, optimizer, evaluator) that automatically builds generalizable, consistent strategy-based few-shot prompts without human-annotated solutions.
Problem
Few-shot CoT prompts with instance-specific, human-annotated solutions lack generalizability across instances and task-level consistency across examples.
Approach
Use LLM agents to inductively derive task-level problem-solving strategies from a few task examples, iteratively execute, optimize, and validate them, then deductively apply the selected strategy to write consistent solutions forming the few-shot prompt.
Key results
- StrategyLLM-SC reaches 38.8% average on MATH via StrategyLLM-ZS vs 34.2% for CoT-SC with GPT-3.5, and 79.2% vs 30.0% on symbolic reasoning (LLC).
- StrategyLLM-SC improves algorithmic reasoning average from 73.7% (CoT-SC) to 85.0%.
- The framework costs $0.08-$0.24 per task with gpt-3.5-turbo-0125 for prompt generation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use StrategyLLM-ZS instead of CoT-SC | GPT-3.5 · MATH Avg | accuracy: 34.2 -> 38.8 | Helped |
| Use StrategyLLM-SC instead of CoT-SC | GPT-3.5 · Commonsense Avg | accuracy: 70.3 -> 72.5 | Helped |
| Use StrategyLLM-SC instead of CoT-SC | GPT-3.5 · Algorithmic Avg | accuracy: 73.7 -> 85.0 | Helped |
| Use StrategyLLM-SC instead of CoT-SC | GPT-3.5 · Symbolic Avg | accuracy: 30.0 -> 79.2 | Helped |
| Use multiple strategies with majority voting instead of one best strategy (ablation) | StrategyLLM · MATH Avg | accuracy: 35.1 -> 37.2 | Helped |
| Use multiple strategies with majority voting instead of one best strategy (ablation) | StrategyLLM · Commonsense Avg | accuracy: 68.0 -> 72.5 | Helped |
| Use multiple strategies with majority voting instead of one best strategy (ablation) | StrategyLLM · Algorithmic Avg | accuracy: 83.4 -> 85.0 | Helped |
| Use multiple strategies with majority voting instead of one best strategy (ablation) | StrategyLLM · Symbolic Avg | accuracy: 78.7 -> 79.2 | Helped |
| Use zero-shot answer selection instead of majority voting across strategies (ablation) | StrategyLLM · MATH Avg | accuracy: 37.2 -> 38.8 | Helped |
| Use zero-shot answer selection instead of majority voting across strategies (ablation) | StrategyLLM · Commonsense Avg | accuracy: 72.5 -> 71.3 | Hurt |
| Use zero-shot answer selection instead of majority voting across strategies (ablation) | StrategyLLM · Algorithmic Avg | accuracy: 85.0 -> 84.0 | Hurt |
| Use zero-shot answer selection instead of majority voting across strategies (ablation) | StrategyLLM · Symbolic Avg | accuracy: 79.2 -> 76.0 | Hurt |
Surprising findings
- Inconsistent prompts mixing top-3 strategies perform much worse than a single consistent strategy (54.3 vs 63.1 average), suggesting consistency matters.
- CoT+Strategy (asking the LLM to devise a strategy at test time) gives limited task-level knowledge and underperforms StrategyLLM.
Limitations
- Effectiveness depends on the LLM's domain knowledge; limited-knowledge domains may yield poor strategies.
- Inference costs are higher than CoT because prompts contain both strategies and solutions.
- Test sets over 200 examples were randomly subsampled to reduce API costs.
Experimental setup
Datasets: MATH StrategyQA Big-Bench Hard Date Understanding Big-Bench Hard Word Sorting Big-Bench Hard Multi-step Arithmetic Last Letter Concatenation
Benchmarks: MATH (7 subjects) StrategyQA Date Understanding Word Sorting Multi-step Arithmetic Last Letter Concatenation (LLC-4/8/16)
Metrics: accuracy
Baselines: Standard Prompting CoT CoT-SC SolutionLLM Plan-and-Solve Prompting CoT+Strategy
Builds on: chain-of-thought prompting self-consistency plan-and-solve prompting
Closest papers by meaning
- Scaling Prompt Synthesis for Large Language Model Reasoning (ICML 2026 · similarity 0.91)
- THOUGHT PROPAGATION: AN ANALOGICAL APPROACH TO COMPLEX REASONING WITH LARGE LANGUAGE MODELS (ICLR 2024 · similarity 0.90)
- PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization (ICLR 2024 · similarity 0.90)
- DOTS: Learning to Reason Dynamically in LLMs via Optimal Reasoning Trajectories Search (ICLR 2025 · similarity 0.90)
- Distilling LLM Agent into Small Models with Retrieval and Code Tools (NeurIPS 2025 · similarity 0.90)
- GReaTer: Gradients Over Reasoning Makes Smaller Language Models Strong Prompt Optimizers (ICLR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.