Research map / Large language models / AI and society
The Defeat of the Winograd Schema Challenge (Abstract Reprint)
Vid Kocijan, Ernest Davis, Thomas Lukasiewicz, Gary Marcus, Leora Morgenstern
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Reviews the history of the Winograd Schema Challenge and argues that large pre-trained transformer language models fine-tuned on WSC-style problems have effectively defeated it (above 90% accuracy by 2019).
Problem
Whether the Winograd Schema Challenge, a commonsense pronoun-disambiguation test proposed by Levesque in 2011, still measures commonsense reasoning given that fine-tuned large language models reach very high accuracy.
Approach
Reviews the history of the WSC and the progression of transformer-based systems that reached over 90% accuracy, and examines what this success means for the challenge's validity as a test of commonsense knowledge.
Key results
- By 2019, AI systems based on large pre-trained transformer language models fine-tuned on WSC-style problems achieved better than 90% accuracy.
Surprising findings
- Fine-tuned transformer language models reaching over 90% on the WSC is presented as defeating a challenge once thought to require commonsense knowledge.
Limitations
- Only the abstract is available; full analysis details are not visible.
Experimental setup
Benchmarks: Winograd Schema Challenge
Metrics: accuracy
Builds on: Winograd Schema Challenge
Closest papers by meaning
- Linguini: A benchmark for language-agnostic linguistic reasoning (NeurIPS 2025 · similarity 0.86)
- Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought (ICLR 2026 · similarity 0.85)
- Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.84)
- RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning (ICML 2025 · similarity 0.84)
- Reasoning Elicitation in Language Models via Counterfactual Feedback (ICLR 2025 · similarity 0.84)
- LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low Resource and Extinct Languages (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
