AI Paper Atlas

Research map / Large language models / Retrieval and knowledge

CREAK: A Dataset for Commonsense Reasoning over Entity Knowledge

Yasumasa Onoe, Michael JQ Zhang, Eunsol Choi, Greg Durrett

NeurIPS 2021 · Large language models › Retrieval and knowledge · approach: Domain-specific QA datasets · 26 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
train on CREAK instead of zero-shot FEVER-KILTRoBERTa-Large · CREAK testaccuracy: 70.2 -> 80.3Improved
scale RoBERTa-Large to T5-3b, in-domainCREAK-trained model · CREAK testaccuracy: 80.3 -> 85.1Improved
scale RoBERTa-Base to RoBERTa-Large, in-domainCREAK-trained model · CREAK testaccuracy: 71.6 -> 80.3Improved
scale RoBERTa-Large to T5-3b, trained on FEVER-KILTzero-shot FEVER-KILT · CREAK testaccuracy: 70.2 -> 76.7Improved
pre-train on FEVER-KILT before CREAK finetuningRoBERTa-Large · CREAK testaccuracy: 80.3 -> 81.1No clear effect
remove entity mentions from claims (ablation)RoBERTa-Large trained on CREAK · CREAK testaccuracy: 80.3 -> 67.3Hurt
add 3 DPR-retrieved Wikipedia passagesRoBERTa-Large trained on CREAK · CREAK testaccuracy: 80.3 -> 84.3Improved
finetune on FEVER-KILT first (retrieval setting)RoBERTa-Large+DPR · CREAK testaccuracy: 84.3 -> 86.8Improved
train on FaVIQ instead of FEVER-KILT zero-shotRoBERTa-Large · CREAK testaccuracy: 70.2 -> 52.6Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy