Research map / Large language models / Language, safety and interpretability
Pre-training Text-to-Text Transformers for Concept-centric Common Sense
Wangchunshu Zhou, Dong-Ho Lee, Ravi Kiran Selvam, Seyeon Lee, Xiang Ren
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply concept-centric generative and contrastive intermediate pre-training | T5-base · CommonsenseQA official dev | accuracy: 61.88($\pm$0.08) -> 63.32($\pm$0.35) | Improved |
| Apply concept-centric generative and contrastive intermediate pre-training | T5-base · OpenBookQA official dev | accuracy: 58.20($\pm$1.0) -> 60.90($\pm$0.4) | Improved |
| Apply concept-centric generative and contrastive intermediate pre-training | T5-base · PIQA official dev | accuracy: 68.14($\pm$0.73) -> 71.01($\pm$0.61) | Improved |
| Apply concept-centric generative and contrastive intermediate pre-training | T5-base · aNLI official dev | accuracy: 61.10($\pm$0.38) -> 63.20($\pm$0.52) | Improved |
| Apply concept-centric pre-training | T5-base · CommonGEN test | BLEU-4: 24.90 -> 26.40 | Improved |
| Add joint training after mixed generative and contrastive warmup (ablation) | CALM (Mix-only) · CommonsenseQA official dev | accuracy: 63.02($\pm$0.47) -> 63.32($\pm$0.35) | Improved |
| Remove the contrastive objective (ablation) | CALM · CommonsenseQA official dev | accuracy: 63.32($\pm$0.35) -> 62.28($\pm$0.36) | Hurt |
| Remove the generative objectives (ablation) | CALM · CommonsenseQA official dev | accuracy: 63.32($\pm$0.35) -> 62.73($\pm$0.41) | Hurt |
| Remove mixed-training warmup before joint training (ablation) | CALM · CommonsenseQA official dev | accuracy: 63.32($\pm$0.35) -> 62.18($\pm$0.48) | Hurt |
| Use COR instead of the original T5 text-infilling objective (ablation) | T5-base · PIQA | accuracy: 68.19 -> 68.77 | Improved |
| Use C2S instead of the original T5 text-infilling objective (ablation) | T5-base · CommonGEN | BLEU-4: 25.10 -> 25.90 | Improved |
| Use generative QA rather than multi-choice QA or true/false contrastive formatting (ablation) | CALM contrastive objective · PIQA | accuracy: 68.82 / 67.81 / 70.67 | Mixed |
Closest papers by meaning
- Self-conditioning Pre-Trained Language Models (ICML 2022 · similarity 0.88)
- The Generalization and Robustness of Transformer-Based Language Models on Commonsense Reasoning (AAAI 2024 · similarity 0.85)
- Self-Supervised Alignment with Mutual Information: Learning to Follow Principles without Preference Labels (NeurIPS 2024 · similarity 0.85)
- Provably Transformers Harness Multi-Concept Word Semantics for Efficient In-Context Learning (NeurIPS 2024 · similarity 0.85)
- Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levels (ICLR 2026 · similarity 0.85)
- Exploring Visual Pretraining for Learning Language Intelligence (CVPR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
