Research map / Large language models / Agents, code and math
TreeEval: Benchmark-Free Evaluation of Large Language Models through Tree Planning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TreeEval, a benchmark-free LLM evaluation where GPT-4 acts as examiner generating questions organized in a tree-planned, adaptive session judged pairwise, achieving top rank correlation with AlpacaEval2.0 using ~45 questions.
Problem
Existing LLM evaluation via static benchmarks suffers from data leakage, and LLM-as-judge pipelines on fixed question sets suffer from biases and inflexibility.
Approach
An LLM examiner dynamically generates questions within topics organized as a tree; the tree is grown adaptively (BFS, with topics extracted via NER from responses) and stops when two models are distinguishable, with pairwise LLM-judge scores aggregated by node importance weights.
Key results
- TreeEval achieves Spearman 0.83 and Kendall 0.73 correlation with AlpacaEval2.0 rankings on six 7B-33B models, the highest among compared methods.
- TreeEval uses on average 45.1 questions per evaluation versus 804 for AlpacaEval and 14,079 for MMLU.
- On stronger models (Yi-34B-Chat baseline, six models), TreeEval reaches Spearman 0.94 and Kendall 0.86 with AlpacaEval2.0 rankings.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Switch tree search from BFS to DFS (ablation) | TreeEval · Model ranking vs AlpacaEval2.0 | Spearman correlation (ρ): 0.83 -> 0.37 | Hurt |
| Add topic generation (Step One) for follow-up questions (ablation) | TreeEval · Model ranking vs AlpacaEval2.0 | Kendall correlation (τ): 0.2 -> 0.73 | Helped |
| Add root-distance weighting to the aggregator (ablation) | TreeEval · Model ranking vs AlpacaEval2.0 | Kendall correlation (τ): 0.6 -> 0.73 | Helped |
| Add topic-origin weighting to the aggregator (ablation) | TreeEval · Model ranking vs AlpacaEval2.0 | Kendall correlation (τ): 0.6 -> 0.73 | Helped |
| Add sibling-variance weighting to the aggregator (ablation) | TreeEval · Model ranking vs AlpacaEval2.0 | Kendall correlation (τ): 0.47 -> 0.73 | Helped |
Surprising findings
- Even a randomly chosen baseline model yields rankings closely resembling those from full pairwise evaluation.
Limitations
- Relies on GPT-4 as examiner and GPT-4 as judge, which may introduce data leakage and bias.
- GPT-4 examiner may be weak outside its areas of expertise.
- Rankings are validated only against AlpacaEval2.0 on a small set of six models; the judge is GPT-4 rather than the ChatGPT judge used by AlpacaEval.
Experimental setup
Benchmarks: MMLU BBH AlpacaEval AlpacaEval2.0 MT-Bench
Metrics: Spearman correlation Kendall correlation number of questions win-rate score variance across runs
Baselines: MMLU BBH AlpacaEval AlpacaEval2.0 MT-Bench
Builds on: Tree-of-Thought AlpacaEval MT-Bench MMLU BBH
Closest papers by meaning
- SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research (AAAI 2024 · similarity 0.92)
- tinyBenchmarks: evaluating LLMs with fewer examples (ICML 2024 · similarity 0.91)
- JudgeLM: Fine-tuned Large Language Models are Scalable Judges (ICLR 2025 · similarity 0.90)
- MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures (NeurIPS 2024 · similarity 0.89)
- DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graph (NeurIPS 2024 · similarity 0.89)
- Reliable and Efficient Amortized Model-based Evaluation (ICML 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.