AI Paper Atlas

Research map / Large language models / Retrieval and knowledge

AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web

Michael Schlichtkrull, Zhijiang Guo, Andreas Vlachos

NeurIPS 2023 · Large language models › Retrieval and knowledge · approach: Domain-specific QA datasets · 7 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Use gold question-answer evidence instead of retrieved evidenceAVeriTeC baseline · AVeriTeC testveracity accuracy at λ=.25: 0.15 -> 0.49Improved
Use gold question-answer evidence instead of retrieved evidenceAVeriTeC baseline · AVeriTeC testjustification METEOR at λ=.25: 0.07 -> 0.28Improved
Use no search or evidence (ablation)AVeriTeC baseline · AVeriTeC testveracity accuracy at λ=.25: 0.15 -> 0.02Hurt
Use no search or evidence (ablation)AVeriTeC baseline · AVeriTeC testveracity macro-F1: .23 -> .17Hurt
Generate one ChatGPT paraphrase per claim and include it in stance-model training (ablation)BERT-large stance detection · AVeriTeC gold evidenceveracity macro-F1: .49 -> .46Hurt
Add predicted veracity as an input to justification generation (ablation)BART-large justification generation · AVeriTeCjustification METEORNo clear effect
Use BLOOM instead of Vicuna for generating search questionsAVeriTeC search-question generation · AVeriTeCretrieval performanceImproved
Use 10 retrieved in-context examples for search-question generationBLOOM search-question generation · AVeriTeCretrieval performanceImproved
Include the claim when generating questions from evidence passages (ablation)BLOOM passage-question generation · AVeriTeCquestion-generation performanceHurt
Use 10 retrieved in-context examples for passage-question generationBLOOM passage-question generation · AVeriTeCquestion-generation performanceImproved
Use BERT-large rather than BLOOM-7b1 or Vicuna-13b for veracity prediction with gold evidenceAVeriTeC veracity prediction · AVeriTeC gold evidenceveracity macro-F1: .43, .48 -> .49Improved
Use BART-large rather than BLOOM-7b1 or Vicuna-13b for justification generationAVeriTeC justification generation · AVeriTeCMETEOR: .28, .23, and .25Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy