Research map / Large language models / Retrieval and knowledge
Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Xiezhi, an auto-updating, discipline-taxonomy-based multiple-choice benchmark (249,587 questions, 516 disciplines) evaluated with 50-option generation-probability ranking across 47 LLMs.
Problem
Existing domain-knowledge benchmarks become outdated quickly as LLM training data grows, lack fine-grained discipline coverage, and 4-option instruction-based multiple-choice evaluation fails for small or non-instruction-tuned LLMs.
Approach
Builds a large, discipline-labeled benchmark auto-updated from new exams and auto-generated survey questions, and scores models by ranking 50 candidate options via generation probability instead of parsing instructed answers.
Key results
- GPT-4 achieves the highest overall MRR on Xiezhi (0.431 in the 0-shot setting) versus 0.337 for the top human average, with LLMs surpassing average human performance in science, engineering, agronomy, and medicine.
- Under the 50-option generation-probability setting, many open-source models score near random guessing (0.089), with bloomz-mt reaching 0.266 on MMLU and 0.379 on Xiezhi-Specialty-English 0-shot.
- All top-10 open-source models are built on llama or bloom and are fine-tuned, and most models do not improve stably with more few-shot demonstrations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the number of demonstration examples from 0-shot to 3-shot | LLMs evaluated on Xiezhi-Specialty Chinese · Xiezhi-Specialty Chinese | MRR: 0.140 -> 0.145 | Helped |
| Increase the number of options per question to 50 | LLMs in the evaluation · Xiezhi | performance | Hurt |
| Increase model parameter count | LLMs across domain benchmarks · Xiezhi | MRR | Mixed |
| Fine-tune on medical domain text | DoctorGLM and Baize-Healthcare · Medical domain | MRR | Helped |
| Fine-tune on medical domain text | DoctorGLM and Baize-Healthcare · Generic domain knowledge | MRR | Hurt |
Surprising findings
- Several open-source models do not improve stably with more few-shot demonstrations, while GPT-4 and ChatGPT do.
- Larger parameter counts do not always yield better performance (e.g., Bloomz-7b1 beats Bloomz-p3; Pythia-1.4b beats larger Pythia models on most benchmarks).
Limitations
- Only Chinese and English versions; limited coverage of other cultures and languages.
- GPT-4 and ChatGPT results obtained via instructions, so true capability may be higher; human baselines derived from exam average scores with known limitations.
- Annotation model lacks fine-grained discipline accuracy and relies on ChatGPT for labels.
Experimental setup
Datasets: Xiezhi-Meta Chinese graduate entrance exam questions Chinese academic surveys and reviews Xiezhi-Train
Benchmarks: Xiezhi-Specialty Xiezhi-Interdiscipline MMLU C-Eval M3KE
Metrics: MRR accuracy
Baselines: MMLU C-Eval M3KE ChatGPT GPT-4
Builds on: MMLU C-Eval M3KE
Closest papers by meaning
- CK12: A Rounded K12 Knowledge Graph Based Benchmark for Chinese Holistic Cognition Evaluation (AAAI 2024 · similarity 0.89)
- KoLA: Carefully Benchmarking World Knowledge of Large Language Models (ICLR 2024 · similarity 0.87)
- INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge (ICLR 2025 · similarity 0.87)
- Neither Valid nor Reliable? Investigating the Use of LLMs as Judges (NeurIPS 2025 · similarity 0.87)
- SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research (AAAI 2024 · similarity 0.87)
- Linguini: A benchmark for language-agnostic linguistic reasoning (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.