Research map / Large language models / Language, safety and interpretability
CFEVER: A Chinese Fact Extraction and VERification Dataset
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CFEVER, a 30,012-claim Chinese fact extraction and verification dataset built from Chinese Wikipedia in FEVER style, and benchmarks a BERT-based baseline, BEVERS, and GPT-3.5/GPT-4 on it.
Problem
Chinese fact verification lacks a large, evidence-annotated benchmark; existing Chinese datasets are small or focus on rumor detection without evidence retrieval.
Approach
Adapt the FEVER construction pipeline to Chinese Wikipedia: writers generate TRUE claims and six variants from sampled sentences and related pages, and annotators label them Supports/Refutes/NEI and select evidence sentences from a fixed fact database.
Key results
- BEVERS reaches 69.73% label accuracy and 64.80% FEVER Score in the full pipeline on the CFEVER test set, versus 52.47% FEVER Score for the authors' BM25-based baseline.
- GPT-4 with 3-shot prompting reaches only 48.40% accuracy on claim verification, showing the dataset is challenging for LLMs.
- Five-way Fleiss' kappa of 0.7934 on 1,936 claims (6.45% of the dataset) indicates high label consistency.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BEVERS instead of BM25 document retrieval | CFEVER document retrieval · CFEVER test | recall: 87.65 -> 92.60 | Helped |
| Use BEVERS sentence retrieval instead of the simple baseline | CFEVER sentence retrieval · CFEVER test | recall: 76.65 -> 86.60 | Helped |
| Use BEVERS claim verification instead of the simple baseline | CFEVER claim verification · CFEVER test | accuracy: 61.17 -> 69.73 | Helped |
| Use BEVERS full pipeline instead of the simple baseline | CFEVER full pipeline · CFEVER test | FEVER Score: 52.47 -> 64.80 | Helped |
| Provide three same-domain labeled examples per class | GPT-3.5 · CFEVER claim verification | accuracy: 43.17 -> 44.20 | Helped |
| Provide three same-domain labeled examples per class | GPT-4 · CFEVER claim verification | accuracy: 47.23 -> 48.40 | Helped |
| Retrieve top 10 rather than top 1 document (ablation) | BM25 document retrieval · CFEVER test | recall: 69.05 -> 87.65 | Helped |
| Retrieve top 10 rather than top 3 documents (ablation) | BM25 document retrieval · CFEVER test | recall: 80.10 -> 87.65 | Helped |
| Retrieve top 10 rather than top 5 documents (ablation) | BM25 document retrieval · CFEVER test | recall: 83.85 -> 87.65 | Helped |
| Retrieve top 10 rather than top 7 documents (ablation) | BM25 document retrieval · CFEVER test | recall: 86.00 -> 87.65 | Helped |
Surprising findings
- GPT-4 zero-shot and 3-shot accuracy (47.23% and 48.40%) stays far below fine-tuned BEVERS, with few-shot giving only slight gains.
- Performance degrades on claims needing evidence from multiple pages, with about 10% of claims challenging to verify.
Limitations
- Dataset is much smaller than English FEVER.
- Most claims (about 88%) require evidence from only a single Wikipedia page, limiting multi-hop reasoning evaluation.
- Human review found 84.4% of sampled claims annotated correctly with correct evidence.
Experimental setup
Datasets: CFEVER Chinese Wikipedia dump (Dec 2022) MNLI
Benchmarks: CFEVER
Metrics: recall accuracy FEVER Score Fleiss kappa
Baselines: BEVERS Stammbach GPT-3.5 GPT-4
Builds on: FEVER BEVERS BM25
Closest papers by meaning
- ViFactCheck: A New Benchmark Dataset and Methods for Multi-Domain News Fact-Checking In Vietnamese (AAAI 2025 · similarity 0.84)
- DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Experts (ICML 2025 · similarity 0.82)
- Unsupervised Pretraining for Fact Verification by Language Model Distillation (ICLR 2024 · similarity 0.82)
- AVerImaTeC: A Dataset for Automatic Verification of Image-Text Claims with Evidence from the Web (NeurIPS 2025 · similarity 0.82)
- BiDeV: Bilateral Defusing Verification for Complex Claim Fact-Checking (AAAI 2025 · similarity 0.82)
- Improving Factual Error Correction by Learning to Inject Factual Errors (AAAI 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.