Research map / Large language models / Agents, code and math

WenMind: A Comprehensive Benchmark for Evaluating Large Language Models in Chinese Classical Literature and Language Arts

NeurIPS 2024 · Large language models › Agents, code and math · approach: Mathematical and structural reasoning benchmarks · 6 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces WenMind, a 4,875-item benchmark of 42 tasks for evaluating LLMs on Chinese Classical Literature and Language Arts, and evaluates 31 models with an LLM-based scoring metric.

Problem

Existing Chinese classical literature benchmarks focus mainly on ancient prose or cover few tasks and formats, so LLM capabilities across ancient prose, poetry, and literary culture remain under-assessed.

Approach

A comprehensive CCLLA benchmark spanning three sub-domains, 42 fine-grained tasks, and three question formats (MCQ, fill-in-the-blank, open-ended), scored mainly with an LLM scorer (ERNIE-3.5) checked against human judgment, with traditional metrics (F1, BLEU) for punctuation and translation.

Key results

• ERNIE-4.0-8K-0329 achieves the top overall score of 64.3 out of 100 on WenMind, while LLaMA2-7B-Chat scores lowest at 13.0.
• The average across 31 evaluated LLMs is 41.2, with over 64% of models scoring below 50.
• Scoring model ERNIE-3.5 agrees with human evaluation about 89.4% of the time.

Measured effects reported in the paper

Surprising findings

• Ancient-Chinese-specialized LLMs (incremental pre-training) underperform general-purpose models, averaging 34.1, suggesting incremental pre-training may cause catastrophic forgetting.
• Traditional metrics are better than the model scorer for punctuation, while the model scorer is preferred for non-unique translation tasks; most models merely rephrase modern text when asked to produce classical Chinese.

Limitations

• Relies on an aligned LLM (ERNIE-3.5) as scorer, which may introduce scoring error despite 89.4% human agreement.
• Some tasks such as punctuation and topic classification serve as general evaluations rather than targeted assessments of practical needs.
• Model scores are from a single run with greedy decoding; scoring stability was checked on only 100 samples.

Experimental setup

Datasets: C2MChn Daizhige Corpus ACLUE WYWEB THU-FSPC

Benchmarks: WenMind

Metrics: model scoring accuracy (rubric points) F1-score BLEU

Baselines: GPT-4 GPT-3.5 ERNIE-4.0 ERNIE-3.5 Qwen1.5 Yi-1.5 InternLM2-Chat-7B Xunzi-Qwen1.5-7B

Builds on: C-Eval CMMLU ACLUE WYWEB

Closest papers by meaning

• NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens (ICLR 2025 · similarity 0.85)
• Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation (AAAI 2024 · similarity 0.85)
• CK12: A Rounded K12 Knowledge Graph Based Benchmark for Chinese Holistic Cognition Evaluation (AAAI 2024 · similarity 0.85)
• INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge (ICLR 2025 · similarity 0.85)
• When LLMs Meet Cunning Texts: A Fallacy Understanding Benchmark for Large Language Models (NeurIPS 2024 · similarity 0.85)
• ScholarGEC: Enhancing Controllability of Large Language Model for Chinese Academic Grammatical Error Correction (AAAI 2025 · similarity 0.85)