Research map / Large language models / Agents, code and math
DynaAct: Large Language Model Reasoning with Dynamic Action Spaces
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DynaAct, which builds a compact per-step action space by greedily maximizing a learned submodular utility-plus-diversity function over LLM-extracted observation sketches, then searches reasoning paths with MCTS on a frozen Llama-3.1-8B.
Problem
Sequential LLM reasoning needs an action space that is both scalable (learned rather than hand-designed) and compact (small yet information-dense per step); existing approaches use either manual action sets or unstructured natural-language search.
Approach
Estimate a proxy full action space from observation sketches extracted by an LLM over a problem corpus, then at each step select a size-m candidate subset by greedily maximizing a submodular function combining a log-sum-exp utility term (learned embedding Q-function via Q-learning) and a min-distance diversity term; MCTS then evaluates the candidates.
Key results
- DynaAct reaches 61.00% on MATH-500 versus 54.20% for rStar and 51.60% for RAP, with 89.16% on GSM8K and 88.31% on ARC-C.
- Ablation removing the submodular function drops MATH-500 accuracy from 61.00 to 52.00 and ARC-C from 88.31 to 85.15.
- On the Level 5 MATH-500 subset, DynaAct achieves 31.34% versus 15.67% for rStar.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DynaAct instead of SC@maj16 | Llama-3.1-8B-Instruct · MMLU | accuracy: 69.66 -> 70.22 | Helped |
| use DynaAct instead of SC@maj16 | Llama-3.1-8B-Instruct · MMLU-Pro | accuracy: 49.36 -> 51.40 | Helped |
| use DynaAct instead of RAP | Llama-3.1-8B-Instruct · GPQA | accuracy: 38.89 -> 39.39 | Helped |
| use DynaAct instead of rStar | Llama-3.1-8B-Instruct · ARC-C | accuracy: 86.43 -> 88.31 | Helped |
| use DynaAct instead of RAP | Llama-3.1-8B-Instruct · GSM8K | accuracy: 87.79 -> 89.16 | Helped |
| use DynaAct instead of rStar | Llama-3.1-8B-Instruct · MATH-500 | accuracy: 54.20 -> 61.00 | Helped |
| add the utility term (ablation) | DynaAct · ARC-C | accuracy: 87.63 -> 88.31 | Helped |
| add the utility term (ablation) | DynaAct · MATH-500 | accuracy: 53.40 -> 61.00 | Helped |
| add the diversity term (ablation) | DynaAct · ARC-C | accuracy: 86.52 -> 88.31 | Helped |
| add the diversity term (ablation) | DynaAct · MATH-500 | accuracy: 53.80 -> 61.00 | Helped |
| add Q-learning to train the embedding function (ablation) | DynaAct · ARC-C | accuracy: 87.80 -> 88.31 | Helped |
| add Q-learning to train the embedding function (ablation) | DynaAct · MATH-500 | accuracy: 55.80 -> 61.00 | Helped |
Surprising findings
- Replacing the submodular diversity term with mean pairwise or mean cosine distance lowers MATH-500 accuracy to 57.80 and 58.20, which the authors attribute to losing submodularity.
- Scaling the proxy action space from 40k to 1M entries leaves accuracy roughly flat (61.0 to 61.8) while latency rises only about 18 seconds.
Limitations
- Reliance on MCTS for action selection is computationally intensive; alternatives such as beam search are not explored.
- Depends on pre-trained Llama models and may struggle with extremely large or highly specialized action spaces.
- No error bars are reported due to computational cost.
Experimental setup
Datasets: Open-Platypus MMLU MMLU-Pro GPQA ARC-C GSM8K MATH-500
Benchmarks: MMLU MMLU-Pro GPQA ARC-C GSM8K MATH-500
Metrics: exact match accuracy relative latency critical step coverage
Baselines: Zero-shot CoT SC@maj16 RAP rStar
Builds on: rap rstar monte carlo tree search q-learning
Closest papers by meaning
- Enhancing Decision-Making of Large Language Models via Actor-Critic (ICML 2025 · similarity 0.90)
- LLM+AL: Bridging Large Language Models and Action Languages for Complex Reasoning About Actions (AAAI 2025 · similarity 0.89)
- T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (ICML 2025 · similarity 0.88)
- Search for Truth from Reasoning: A Dynamic Representation Editing Framework for Steering LLM Trajectories (ICML 2026 · similarity 0.88)
- DOTS: Learning to Reason Dynamically in LLMs via Optimal Reasoning Trajectories Search (ICLR 2025 · similarity 0.88)
- Improving Reasoning Performance in Large Language Models via Representation Engineering (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.