Research map / Large language models / Retrieval and knowledge
WinoGrande: An Adversarial Winograd Schema Challenge at Scale
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, Yejin Choi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Filter predictable instances using AfLite instead of random data reduction | WinoGrande · WinoGrande embeddings | KL divergence: 2.53 → 2.51; AfLite → 0.12 | Improved |
| Filter twins by PMI differences instead of AfLite | WinoGrande · WinoGrande embeddings | KL divergence: → 2.42 | Improved |
| Use AfLite filtering rather than PMI filtering | WinoGrande · WinoGrande embeddings | KL divergence: 2.42 → 0.12 | Improved |
| Use only local context around the blank instead of the full sentence (ablation) | BERT and RoBERTa · WinoGrande debiased test | accuracy (%): 64.9 -> 51.9; 79.1 -> 50.0 | Hurt |
| Increase RoBERTa training data from XS to XL | RoBERTa · WinoGrande debiased test | accuracy (%): 50.4 -> 79.1 | Improved |
| Fine-tune RoBERTa on WinoGrande before WSC evaluation, rather than DPR | RoBERTa · WSC test | accuracy (%): 83.1 -> 90.1 | Improved |
| Fine-tune RoBERTa on WinoGrande before PDP evaluation, rather than DPR | RoBERTa · PDP test | accuracy (%): 86.3 -> 87.5 | Improved |
| Sequentially fine-tune on WinoGrande before SuperGLUE-WSC | RoBERTa · SuperGLUE-WSC test | accuracy (%): 83.6 -> 85.6 | Improved |
| Fine-tune on WinoGrande before DPR evaluation | RoBERTa · DPR test | accuracy (%): 91.7 -> 93.1 | Improved |
| Fine-tune RoBERTa on WinoGrande before KnowRef evaluation, rather than DPR | RoBERTa · KnowRef test | accuracy (%): 84.2 -> 85.6 | Improved |
| Sequentially fine-tune on WinoGrande before COPA | RoBERTa · COPA test | accuracy (%): 86.4 -> 90.6 | Improved |
| Fine-tune RoBERTa on WinoGrande rather than DPR for Winogender | RoBERTa · Winogender | absolute gender accuracy gap: 1.6, 0.9 -> 0.8, 0.8 | Improved |
Closest papers by meaning
- The Generalization and Robustness of Transformer-Based Language Models on Commonsense Reasoning (AAAI 2024 · similarity 0.87)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.87)
- Scaling Prompt Synthesis for Large Language Model Reasoning (ICML 2026 · similarity 0.86)
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.86)
- Underspecification in Language Modeling Tasks: A Causality-Informed Study of Gendered Pronoun Resolution (AAAI 2024 · similarity 0.86)
- SEED-Bench: Benchmarking Multimodal Large Language Models (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
