AI Paper Atlas

Research map / Large language models / AI and society

The Defeat of the Winograd Schema Challenge (Abstract Reprint)

Vid Kocijan, Ernest Davis, Thomas Lukasiewicz, Gary Marcus, Leora Morgenstern

AAAI 2024 · Large language models › AI and society · approach: Data commons and human baselines · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted summary. Written by AI from the paper's abstract only. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Reviews the history of the Winograd Schema Challenge and argues that large pre-trained transformer language models fine-tuned on WSC-style problems have effectively defeated it (above 90% accuracy by 2019).

Problem

Whether the Winograd Schema Challenge, a commonsense pronoun-disambiguation test proposed by Levesque in 2011, still measures commonsense reasoning given that fine-tuned large language models reach very high accuracy.

Approach

Reviews the history of the WSC and the progression of transformer-based systems that reached over 90% accuracy, and examines what this success means for the challenge's validity as a test of commonsense knowledge.

Key results

Surprising findings

Limitations

Experimental setup

Benchmarks: Winograd Schema Challenge

Metrics: accuracy

Builds on: Winograd Schema Challenge

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy