Research map / Large language models / Retrieval and knowledge
FELM: Benchmarking Factuality Evaluation of Large Language Models
Shiqi Chen, Yiran Zhao, Jinghan Zhang, I-Chun Chern, Siyang Gao, Pengfei Liu, Junxian He
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add retrieved reference-document content (ablation) | ChatGPT segment-based evaluator · FELM All | F1: 4.9 -> 15.7 | Improved |
| Add retrieved reference-document content (ablation) | GPT-4 segment-based evaluator · FELM All | F1: 35.4 -> 48.3 | Improved |
| Add chain-of-thought prompting (ablation) | GPT-4 factuality evaluators · FELM, nearly all reported domains | F1: 35.4 -> 42.0 | Improved |
| Add chain-of-thought prompting (ablation) | ChatGPT factuality evaluators · FELM, all reported settings | F1: 4.9 -> 3.8 | Hurt |
| Use claim-based instead of segment-based evaluation (ablation) | ChatGPT factuality evaluators · FELM All | F1: 4.9 -> 11.8 | Improved |
| Use claim-based instead of segment-based evaluation (ablation) | GPT-4 vanilla evaluator · FELM All | F1: 35.4 -> 33.1 | Hurt |
Closest papers by meaning
- Long-form factuality in large language models (NeurIPS 2024 · similarity 0.91)
- Fine-Tuning Language Models for Factuality (ICLR 2024 · similarity 0.91)
- Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluation (NeurIPS 2025 · similarity 0.90)
- Is Factuality Enhancement a Free Lunch For LLMs? Better Factuality Can Lead to Worse Context-Faithfulness (ICLR 2025 · similarity 0.90)
- The FACTS Leaderboard: A Comprehensive Benchmark for Large Language Model Factuality (NeurIPS 2026 · similarity 0.90)
- KoLA: Carefully Benchmarking World Knowledge of Large Language Models (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
