Research map / Large language models / Agents, code and math
DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DyVal, a dynamic evaluation protocol that generates reasoning test samples on the fly from DAGs with controllable complexity, and shows LLMs degrade with difficulty and that DyVal data helps fine-tuning.
Problem
Static LLM benchmarks suffer from data contamination and fixed complexity, so they may not gauge advancing LLM reasoning capabilities.
Approach
A general dynamic evaluation protocol (generation G, constraints C, description function F) instantiated with graph-informed DAGs: tree-based DAGs for arithmetic/logic and general DAGs for reachability/max-sum-path, with complexity controlled by depth, width, nodes, links and random descriptions.
Key results
- GPT-4 performs best across DyVal tasks, with GPT-3.5-Turbo closely behind; GPT-3.5-Turbo's arithmetic performance drops by 23% from D1 to D4.
- Flan-T5-large, phi-1.5, WizardMath-13B and Xwin-13B score essentially 0 on DyVal tasks despite strong reported results on static benchmarks.
- Fine-tuning Llama2-13B-chat on DyVal-generated data improves performance on 6 existing benchmarks (GSM8K, SVAMP, FOLIO, RACO, DP, LCS).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase arithmetic task complexity from D1 to D4 | GPT-3.5-Turbo · DyVal arithmetic | accuracy: 23% | Hurt |
| Increase complexity constraints, especially tree depth (ablation) | GPT-3.5-Turbo · DyVal arithmetic, Boolean logic, and deductive logic | accuracy | Hurt |
| Use Automatic Prompt Engineering (APE) (ablation) | LLMs evaluated on DyVal · DyVal linear equation | accuracy: 10% | Helped |
| Use Automatic Prompt Engineering (APE) (ablation) | LLMs evaluated on DyVal · DyVal deductive and abductive logic | accuracy | Hurt |
| Increase Llama2 model size (ablation) | Llama2 · DyVal arithmetic, Boolean logic, and reachability D1 | accuracy | Helped |
| Fine-tune Llama2-13B-chat on DyVal-generated data | Llama2-13B-chat · GSM8K, SVAMP, FOLIO, RACO, DP, and LCS | performance | Helped |
Surprising findings
- Phi-1.5, WizardMath-13B and Xwin-13B score near zero on DyVal despite strong claimed results on static benchmarks.
- Llama2-13B-chat's reachability accuracy paradoxically rises with difficulty because it mostly guesses 'True' and often gives no response.
Limitations
- Currently limited to reasoning tasks; extending to new tasks requires designing a new generation algorithm.
- Only a limited number of test samples (500 per dataset) due to resource constraints.
- Fine-tuning explored only on a limited set of models and datasets.
Experimental setup
Datasets: GSM8K SVAMP FOLIO RACO DP LCS
Benchmarks: DyVal arithmetic DyVal linear equation DyVal boolean logic DyVal deductive logic DyVal abductive logic DyVal reachability DyVal max sum path GSM8K SVAMP FOLIO RACO DP
Metrics: accuracy relative precision
Baselines: Flan-T5-large phi-1.5 WizardMath-13B Xwin-13B Llama2-13B-chat Vicuna-13B-v1.3 GPT-3.5-Turbo GPT-4
Builds on: chain-of-thought least-to-most prompting automatic prompt engineering skills-in-context prompting
Closest papers by meaning
- DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graph (NeurIPS 2024 · similarity 0.92)
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025 · similarity 0.91)
- ThinkBench: Dynamic Out-of-Distribution Evaluation for Robust LLM Reasoning (NeurIPS 2025 · similarity 0.91)
- Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.90)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.90)
- tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.