AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

Pre-training Text-to-Text Transformers for Concept-centric Common Sense

Wangchunshu Zhou, Dong-Ho Lee, Ravi Kiran Selvam, Seyeon Lee, Xiang Ren

ICLR 2021 · Large language models › Language, safety and interpretability · approach: Sign and cross-lingual translation · 35 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Apply concept-centric generative and contrastive intermediate pre-trainingT5-base · CommonsenseQA official devaccuracy: 61.88($\pm$0.08) -> 63.32($\pm$0.35)Improved
Apply concept-centric generative and contrastive intermediate pre-trainingT5-base · OpenBookQA official devaccuracy: 58.20($\pm$1.0) -> 60.90($\pm$0.4)Improved
Apply concept-centric generative and contrastive intermediate pre-trainingT5-base · PIQA official devaccuracy: 68.14($\pm$0.73) -> 71.01($\pm$0.61)Improved
Apply concept-centric generative and contrastive intermediate pre-trainingT5-base · aNLI official devaccuracy: 61.10($\pm$0.38) -> 63.20($\pm$0.52)Improved
Apply concept-centric pre-trainingT5-base · CommonGEN testBLEU-4: 24.90 -> 26.40Improved
Add joint training after mixed generative and contrastive warmup (ablation)CALM (Mix-only) · CommonsenseQA official devaccuracy: 63.02($\pm$0.47) -> 63.32($\pm$0.35)Improved
Remove the contrastive objective (ablation)CALM · CommonsenseQA official devaccuracy: 63.32($\pm$0.35) -> 62.28($\pm$0.36)Hurt
Remove the generative objectives (ablation)CALM · CommonsenseQA official devaccuracy: 63.32($\pm$0.35) -> 62.73($\pm$0.41)Hurt
Remove mixed-training warmup before joint training (ablation)CALM · CommonsenseQA official devaccuracy: 63.32($\pm$0.35) -> 62.18($\pm$0.48)Hurt
Use COR instead of the original T5 text-infilling objective (ablation)T5-base · PIQAaccuracy: 68.19 -> 68.77Improved
Use C2S instead of the original T5 text-infilling objective (ablation)T5-base · CommonGENBLEU-4: 25.10 -> 25.90Improved
Use generative QA rather than multi-choice QA or true/false contrastive formatting (ablation)CALM contrastive objective · PIQAaccuracy: 68.82 / 67.81 / 70.67Mixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy