Research map / Large language models / Agents, code and math
DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graph
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DARG, which extends static benchmarks by perturbing LLM-extracted reasoning graphs to create controllable-complexity, linguistically diverse test data verified by a code-augmented LLM, and evaluates 15 LLMs across math, social, spatial, and symbolic reasoning.
Problem
Static LLM benchmarks suffer from data contamination and fixed complexity, while prior dynamic evaluation methods either are rigid template-based or use unstable LLM prompting with poor label control.
Approach
Extract reasoning graphs from benchmark items, perturb graph structure (width, depth) and node values with rule-based functions to control complexity, then decode back to text with an LLM using in-context exemplars, and verify labels with a code-interpreter LLM agent.
Key results
- Claude-3-Opus accuracy on GSM8K drops by 54.2% as depth increases by 4 (from 95% original), per the paper's example.
- All models except GPT-4 Turbo show over 40-point drop in positive accuracy on BBH Navigate when nodes increase by 16.
- GPT-4 Turbo and Gemini-1.5-Pro show highest over-sensitivity (avoidance) on BBQ; fine-tuning Mistral-7B and Llama-2-7B on DARG data outperforms equal-size GSM8K training data (mean paired t-test p=0.022).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase GSM8K reasoning-graph depth by +4 | LLMs evaluated with DARG · GSM8K | accuracy: 95% accuracy on the original test set; drops by 54.2% | Hurt |
| Increase numerical complexity to +8 | LLMs evaluated with DARG · GSM8K | accuracy: 83.8 -> 13.8 | Hurt |
| Increase reasoning-graph width to +4 | LLMs evaluated with DARG · GSM8K | accuracy: 83.8 -> 51.5 | Hurt |
| Increase the number of attribute pairs | LLMs evaluated with DARG · BBQ | overall accuracy | Hurt |
| Increase the number of attribute pairs | LLMs evaluated with DARG · BBQ | bias score | Hurt |
| Increase navigation reasoning steps by +16 | LLMs evaluated with DARG · BBH Navigate | positive accuracy: over 40 percent decline for all models except GPT-4 Turbo | Hurt |
| Increase the number of input brackets | LLMs evaluated with DARG · BBH Dyck Language | exact match accuracy | Mixed |
| Increase the number of output brackets | LLMs evaluated with DARG · BBH Dyck Language | exact match accuracy: 22.8 -> 15.6 | Hurt |
| Fine-tune on DARG-generated data instead of an equivalent amount of GSM8K training data | Mistral-7B and Llama2-7B · GSM8K test set evolved by DARG | accuracy | Helped |
Surprising findings
- GPT-4 Turbo and Gemini-1.5-Pro, the strongest models on BBQ overall accuracy, show more over-sensitivity to protected-group content, often choosing 'Cannot be determined' even with clear evidence.
- On BBH Dyck, GPT-4 Turbo and Mixtral-8x22B accuracy increases at small input sizes (4 and 8 nodes) before declining.
Limitations
- Evaluated only one representative dataset per reasoning task due to limited resources.
- Graph extraction and data generation rely heavily on closed-source LLMs such as GPT-4; open-source generation not explored.
- Fine-tuning experiments limited to two 7B models on math data.
Experimental setup
Datasets: GSM8K BBQ BBH Navigate BBH Dyck Language
Benchmarks: GSM8K BBQ BBH Navigate BBH Dyck Language
Metrics: Pass@1 accuracy Complexity-Induced Accuracy Retention Rate (CIARR) bias score overall avoidance rate positive/negative accuracy exact match accuracy
Baselines: DyVal 2 Benchmark Self-Evolving GPT-3.5 Turbo
Builds on: DyVal Chain-of-Thought prompting Least-to-Most prompting LitGPT LoRA
Closest papers by meaning
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks (ICLR 2024 · similarity 0.92)
- ThinkBench: Dynamic Out-of-Distribution Evaluation for Robust LLM Reasoning (NeurIPS 2025 · similarity 0.89)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.89)
- TreeEval: Benchmark-Free Evaluation of Large Language Models through Tree Planning (AAAI 2025 · similarity 0.89)
- DyCodeEval: Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025 · similarity 0.89)
- Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.