Research map / Large language models / Agents, code and math

RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning

ICML 2025 · Large language models › Agents, code and math · approach: Inductive and abstract reasoning evaluation · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces RULEBREAKERS, a templated dataset of 25,600 minimally differing rulebreaker/non-rulebreaker pairs, and shows seven LLMs over-rigidly apply formal logic (MT/DS), failing to reject factually contradicted conclusions.

Problem

LLMs are often evaluated and enhanced with formal logic, but formal rules can yield conclusions that humans reject because they contradict common sense or factual knowledge ("rulebreakers"); this knowledge-informed, human-like reasoning has not been systematically tested in NLP.

Approach

Template-generated minimal pairs where the same modus tollens or disjunctive syllogism structure yields a conclusion that is either factually inconsistent with the premises (rulebreaker) or consistent (non-rulebreaker), with paired accuracy and token-probability confidence metrics; analyses link failures to entity familiarity and attention to the factual premise.

Key results

• Only Meta-Llama-3-8B-Instruct exceeds 0.6 paired accuracy (0.609 on the base setup); GPT-4o scores 15.06% paired accuracy under the baseline phrasing while achieving 100% on non-rulebreakers.
• All six open models are more confident in 'yes' for non-rulebreakers than for rulebreakers (p<0.0001), suggesting latent discrimination ability.
• Alternative phrasing with prefixed premises raised Phi-3-medium paired accuracy from 29.19% to 44.66% and GPT-4o from 15.06% to 31.92%.

Measured effects reported in the paper

Surprising findings

• GPT-4o underperforms the smaller Phi-3-mini-128k-Instruct in paired accuracy, and Gemma-2-27b-it and GPT-4o achieve perfect non-rulebreaker accuracy while scoring worst on rulebreakers.
• Model familiarity with entities does not guarantee good performance; Gemma-2-27b-it is the most familiar with entities yet among the worst on paired accuracy.

Limitations

• Only two logical rules (modus tollens and disjunctive syllogism) and two entity-pair types are covered.
• Only medium-sized, instruction-tuned open models were evaluated in full; GPT-4o was evaluated only in a limited setting without probability access.
• Analysis links to familiarity and attention are mixed and not uniformly consistent across models.

Experimental setup

Datasets: WikiData country-capital pairs ConceptNet type-instance pairs Popular Names by Country Dataset

Benchmarks: RULEBREAKERS

Metrics: paired accuracy rulebreaker accuracy non-rulebreaker accuracy model confidence (Pi+ values) attention score ratio input x gradient score ratio

Baselines: GPT-4o

Builds on: mental model theory ProntoQA LogicNLI

Closest papers by meaning

• Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement (ICLR 2024 · similarity 0.90)
• Are Language Models Efficient Reasoners? A Perspective from Logic Programming (NeurIPS 2025 · similarity 0.89)
• Unveiling the Magic of Code Reasoning through Hypothesis Decomposition and Amendment (ICLR 2025 · similarity 0.89)
• Does Reasoning Emerge? Examining the Probabilities of Causation in Large Language Models (NeurIPS 2024 · similarity 0.89)
• RATT: A Thought Structure for Coherent and Correct LLM Reasoning (AAAI 2025 · similarity 0.88)
• Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models (ICLR 2025 · similarity 0.88)