Research map / Large language models / Agents, code and math
Creativity or Brute Force? Using Brainteasers as a Window into the Problem-Solving Abilities of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the Braingle Brainteaser benchmark of expert-written math and logic puzzles and uses it to probe LLM reasoning strategies (brute force vs. creative insight), hint use, self-correction, rewriting, and step breakdown.
Problem
Accuracy alone says little about how LLMs solve problems; the paper asks whether models use creative insight or brute-force search on brainteasers, and how they handle related sub-tasks like rewriting, self-correction, and hints.
Approach
Uses narrative brainteasers with multiple possible solution routes (short creative insight vs. long brute force), with human solutions and hints, to evaluate not only correctness but solution strategy via o3-based judging and step-count analysis.
Key results
- OpenAI o3 reached 88.8% on the Math set with hints and math prompting, but only 80.5% on the top-50 hardest problems.
- Rewriting the 30 hardest rewritable math puzzles into competition style raised o3 accuracy from 56.7% to 73.3% (p=0.023).
- o3's judgments agreed with human annotators at 99.3% (correctness) and 97% (creative vs brute-force) on 100 solutions.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add a problem-specific hint to CoT prompting | OpenAI o3 · Braingle Math | accuracy (%): 79.6 -> 82.8 | Helped |
| Add a problem-specific hint to Math Prompt | OpenAI o3 · Braingle Math | accuracy (%): 79.6 -> 81.2 | Helped |
| Add a problem-specific hint to CoT prompting | OpenAI o3 · Braingle Logic | accuracy (%): 68.4 -> 70.0 | Helped |
| Add a problem-specific hint to Math Prompt | OpenAI o3 · Braingle Logic | accuracy (%): 71.2 -> 74.4 | Helped |
| Rewrite narrative statements into mathematical-format problems | OpenAI o3 · Top 30 rewritable Braingle Math problems | accuracy (%): 56.7% to 73.3% | Helped |
| Rewrite narrative statements into mathematical-format problems | deepseek-reasoner · Top 30 rewritable Braingle Math problems | accuracy (%): 50.0% to 63.3% | Helped |
| Rewrite narrative statements into mathematical-format problems | DeepSeek-R1-Distill-LLaMA-70B · Top 30 rewritable Braingle Math problems | accuracy (%): 23.3% to 33.3% | No clear effect |
| Use a Math prompt rather than CoT prompting | LLMs · Braingle Math and Braingle Logic | brute-force usage | Helped |
| Combine mathematical framing with a hint | LLMs · Braingle Math and Braingle Logic | brute-force usage | Helped |
| Increase model size and capability | Evaluated LLMs · Braingle Brainteaser | brute-force usage | Helped |
Surprising findings
- Models often accept a false premise: over 60% of the time they wrongly accepted that a correct human solution was incorrect when swayed by the prompt.
- Simply instructing models to avoid brute force yielded minimal benefit, while structured hints gave consistent gains.
Limitations
- Findings are derived mainly from one source (Braingle) and focus on math/logic puzzles, limiting generalizability.
- Possible data contamination since problems come from a public website; evidence is anecdotal.
- Single run per setting due to API cost, so no error bars; reliance on o3 as judge may introduce bias.
Experimental setup
Datasets: Braingle Math (242 problems after QC) Braingle Logic (236 problems after QC)
Benchmarks: Braingle Brainteaser Math Braingle Brainteaser Logic
Metrics: solution accuracy brute-force rate step counts (creative vs rudimentary) rewriting accuracy self-correction error modes
Baselines: DeepSeek R1 Distill Qwen 1.5B DeepSeek R1 Distill Qwen 14B DeepSeek R1 Distill Llama 70B DeepSeek-V3 DeepSeek-R1 Gemini 2.5 Flash OpenAI o3
Builds on: chain-of-thought prompting
Closest papers by meaning
- Assessing the Creativity of LLMs in Proposing Novel Solutions to Mathematical Problems (AAAI 2025 · similarity 0.89)
- Eliciting Reasoning in Language Models with Cognitive Tools (NeurIPS 2025 · similarity 0.89)
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity (NeurIPS 2025 · similarity 0.89)
- THOUGHT PROPAGATION: AN ANALOGICAL APPROACH TO COMPLEX REASONING WITH LARGE LANGUAGE MODELS (ICLR 2024 · similarity 0.89)
- Large Language Models Think Too Fast To Explore Effectively (NeurIPS 2025 · similarity 0.88)
- OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.