Research map / Large language models / Reasoning and chain of thought

HardcoreLogic: Challenging Large Reasoning Models with Long-tail Logic Puzzle Games

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces HardcoreLogic, a benchmark of over 5,000 long-tail logic puzzles across 10 games built by increasing complexity, uncommon elements, and unsolvable variants, and shows that frontier LRMs degrade substantially on it.

Problem

Whether large reasoning models genuinely apply logical rules flexibly, or instead rely on memorized canonical puzzle forms, is unclear because existing logic-puzzle corpora focus on canonical variants (e.g., 9x9 Sudoku).

Approach

Systematically transform canonical logic puzzles along three long-tail dimensions (increased complexity, uncommon elements/rules, and unsolvable puzzles), using solver statistics (Z3, Dijkstra) and perplexity to verify added difficulty, and evaluate many LRMs plus an error taxonomy.

Key results

• Models show significant accuracy drops from Original to HardcoreLogic; Kimi-K2-Instruct shows the largest drop, and gpt-oss-120b is best among open-source models while GPT-5 is best among closed-source.
• Increased search space (IC1) is the dominant difficulty source in regression analysis, e.g., Binario accuracy coefficient of -74.09 for IC1.
• Factual errors are the most prevalent error type; stronger models show brute-force errors, and weaker models like MiniMax-M1 output 'unsolvable' without justification.

Surprising findings

• Weaker models such as MiniMax-M1 often claim unsolvability when they simply fail to find a solution, rather than through genuine recognition.
• Closed-source models generally outperform open-source models on unsolvable puzzles.

Limitations

• Closed-source models evaluated on a 600-case subsample due to cost, and per-game analysis uses open-source models only.
• Closed-source model results are not exactly reproducible due to stochastic black-box serving.
• Some transformations (e.g., Sudoku with additional constraints) were reclassified after difficulty comparison, making categorization partly post hoc.

Experimental setup

Datasets: Enigmata ZebraLogic

Benchmarks: HardcoreLogic Original (Enigmata + ZebraLogic + Hanoi subset)

Metrics: accuracy error type distribution

Baselines: GPT-5 gpt-oss-120b Qwen3-235B-A22B-Thinking-2507 Kimi-K2-Instruct MiniMax-M1-40k DeepSeek-R1-0528 Gemini 2.5 Pro Claude Sonnet 4

Builds on: Enigmata ZebraLogic

Closest papers by meaning

• PuzzleWorld: A Benchmark for Multimodal, Open-Ended Reasoning in Puzzlehunts (ICLR 2026 · similarity 0.89)
• ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning (ICML 2025 · similarity 0.89)
• Causal language modeling can elicit search and reasoning capabilities on logic puzzles (NeurIPS 2024 · similarity 0.88)
• Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzles (NeurIPS 2025 · similarity 0.88)
• GTBench: Uncovering the Strategic Reasoning Capabilities of LLMs via Game-Theoretic Evaluations (NeurIPS 2024 · similarity 0.87)
• The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity (NeurIPS 2025 · similarity 0.87)