Research map / Large language models / Language, safety and interpretability
BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Information
Mehran Kazemi, Quan Yuan, Deepti Bhatia, Najoung Kim, Xin Xu, Vaiva Imbrasaite, Deepak Ramachandran
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase reasoning depth from 1 to 3 | finetuned BERT and T5 · BoardgameQA | accuracy | Hurt |
| few-shot prompting instead of tuning | PaLM 62B/540B · BoardgameQA | conflict F1 | Hurt |
| scale PaLM from 62B to 540B | PaLM few-shot · BoardgameQA | accuracy | Improved |
| increase conflict probability from none to high (0.0, 0.2, 0.5, 0.8) | all models · BoardgameQA | accuracy | Hurt |
| shift conflicts toward Type1 (p=0.2 vs 0.8 Type1) | all models · BoardgameQA | accuracy | Improved |
| bias dataset toward a single conflict type | tuned models · BoardgameQA | accuracy | Improved |
| increase required missing knowledge (p_MissInfo 0.2 to 0.8) | finetuned models · BoardgameQA | accuracy | Hurt |
| increase required missing knowledge (p_MissInfo 0.2 to 0.8) | prompt-tuned and few-shot PaLM · BoardgameQA | accuracy | No clear effect |
| add 1 distractor per step | tuned models · BoardgameQA | accuracy | No clear effect |
| add 2 distractors per step | tuned models · BoardgameQA | accuracy | Hurt |
| increase distractors | few-shot PaLM · BoardgameQA | accuracy | Hurt |
| binary (proved/disproved) version of the dataset | T5 · BoardgameQA depth 2 | accuracy | Improved |
Closest papers by meaning
- RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning (ICML 2025 · similarity 0.88)
- AbstentionBench: Reasoning LLMs Fail on Unanswerable Questions (NeurIPS 2025 · similarity 0.88)
- Linguini: A benchmark for language-agnostic linguistic reasoning (NeurIPS 2025 · similarity 0.88)
- Learning to Reason via Mixture-of-Thought for Logical Reasoning (ICLR 2026 · similarity 0.87)
- Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.87)
- Argument Graph Uncertainty: Quantifying Uncertainty from the Logical Structure of Reasoning Chains (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
