Research map / Large language models / Retrieval and knowledge
Evaluating Commonsense in Pre-Trained Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use a bidirectional contextualizer instead of GPT2-medium | commonsense sentence scoring · CA, WSC, SM, SMR, SWAG, HellaSwag, ARCT1, ARCT2 | accuracy: RoBERTa-large: 0.956, 0.625, 0.750, 0.423, 0.691, 0.414, 0.500, 0.537; GPT2-medium: 0.885, 0.569, 0.746, 0.385, 0.597, 0 | Mixed |
| Use BERT-large rather than GPT2-medium | commonsense sentence scoring · SM | accuracy: 0.746 -> 0.694 | Hurt |
| Increase model size from GPT2-base to GPT2-medium | GPT2 · CA, WSC, SM, SMR, SWAG, HellaSwag, ARCT1, ARCT2 | accuracy: CA 0.787 -> 0.885; WSC 0.512 -> 0.569; SM 0.706 -> 0.746; SMR 0.355 -> 0.385; SWAG 0.522 -> 0.597; HellaSwag 0.300 -> 0. | Mixed |
| Train with the larger RoBERTa corpus and its associated training design | RoBERTa versus GPT2-base · CA, WSC, SM, SMR, SWAG, HellaSwag, ARCT1, ARCT2 | accuracy: RoBERTa-base: 0.956, 0.625, 0.750, 0.423, 0.691, 0.414, 0.500, 0.537; GPT2-base: 0.787, 0.512, 0.706, 0.355, 0.522, 0.30 | Helped |
| Increase the number of required inference steps | pre-trained contextualized models · CATs tasks | accuracy | Hurt |
| Evaluate on dual samples with additions, deletions, or substitutions | pre-trained contextualized models · dual-instance robustness test | consistency: Add: 0.16–0.36; Del: 0.15–0.39; Sub: 0.21–0.44; RANDOM: 0.50 | Hurt |
| Swap word positions in dual samples | pre-trained contextualized models · dual-instance robustness test | consistency: 0.36–0.55; RANDOM: 0.50 | Mixed |
| Use pre-trained language models rather than random choice | CATs commonsense tasks · all tasks except ARCT | accuracy | Helped |
| Use RoBERTa-large rather than the strongest other model | commonsense sentence scoring · ARCT1 | accuracy: 0.563 (XLNet-large) vs 0.543 (RoBERTa-large) | Hurt |
Closest papers by meaning
- Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering (AAAI 2021 · similarity 0.90)
- The Generalization and Robustness of Transformer-Based Language Models on Commonsense Reasoning (AAAI 2024 · similarity 0.88)
- Global PIQA: Evaluating Commonsense Reasoning Across 100+ Languages and Cultures (NeurIPS 2026 · similarity 0.87)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.86)
- Multimodal Commonsense Knowledge Distillation for Visual Question Answering (Student Abstract) (AAAI 2025 · similarity 0.86)
- Potemkin Understanding in Large Language Models (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.