Research map / Large language models / Retrieval and knowledge
CK12: A Rounded K12 Knowledge Graph Based Benchmark for Chinese Holistic Cognition Evaluation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds CK12, a Chinese K12 benchmark of 39,452 questions organized by a teacher-built knowledge graph (584 level-1, 1,989 level-2 points), and evaluates 8 LLMs under four prompt modes plus shuffled/distractor choice variants and step-level CoT metrics.
Problem
Existing Chinese LLM benchmarks draw from exam papers and cover K12 knowledge points incompletely; they also lack reasoning-step evaluation and are prone to data contamination.
Approach
A multi-level K12 knowledge graph built by teachers defines coverage; choice questions are made harder via option shuffling and injected distractor options to reduce contamination, and a math subset is scored with reasoning-step metrics (Golovneva et al. style) for interpretability.
Key results
- GPT-4 achieves the highest accuracy in most settings, e.g., 70.1 / 61.3 (Arts/Sciences) on single-choice zero-shot.
- Shuffling options has minimal effect (about 1 point), while adding distractors causes significant drops for all models, especially multiple-choice.
- Arts accuracy exceeds Science accuracy by more than 10 percentage points across models, suggesting weaker reasoning than memory in LLMs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| shuffle answer options | CK12 choice-question evaluation · CK12 single-choice and multiple-choice questions | accuracy | No clear effect |
| add distractor options | CK12 choice-question evaluation · CK12 single-choice and multiple-choice questions | accuracy | Hurt |
| use CoT prompting instead of the corresponding non-CoT prompt | most tested models · CK12 benchmark, multiple question types | accuracy | Hurt |
| use few-shot prompting instead of zero-shot prompting | most finetuned models · CK12 benchmark | accuracy | Hurt |
Surprising findings
- Few-shot prompting often hurts fine-tuned chat models, and CoT prompting does not reliably improve results for many subjects.
- ChatGLM2 gets a high reasoning-alignment score despite low accuracy, while InternLM has high accuracy but low alignment.
Limitations
- Few-shot evaluation of GPT-4 was pending due to token limits, so some GPT-4 cells are missing.
- Reasoning-step metrics are described as only relative reference scores with limited absolute meaning.
- Questions come from a private bank, so reproducibility of the full set is limited.
Experimental setup
Datasets: CK12 question bank (private, teacher-curated)
Benchmarks: CK12
Metrics: accuracy faithfulness-step source-consistency reasoning alignment hallucination redundancy
Baselines: GPT-4 ChatGPT InternLM ChatGLM2 Chinese-Alpaca-plus Baichuan Moss Llama
Builds on: Golovneva et al. 2022 ROSCOE reasoning step metrics Chain-of-thought prompting C-Eval Bloom's taxonomy
Closest papers by meaning
- Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation (AAAI 2024 · similarity 0.89)
- MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models (AAAI 2026 · similarity 0.88)
- KoLA: Carefully Benchmarking World Knowledge of Large Language Models (ICLR 2024 · similarity 0.88)
- BMMR: A Large-Scale Bilingual Multimodal Multi-Discipline Reasoning Dataset (NeurIPS 2025 · similarity 0.88)
- K12-KGraph: A Curriculum-Aligned Knowledge Graph for Benchmarking and Training Educational LLMs (NeurIPS 2026 · similarity 0.88)
- LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.