Research map / Large language models / Reasoning and chain of thought
MP: Endowing Large Language Models with Lateral Thinking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes metacognitive prompting (MP), a three-step planning, reasoning, and reflecting prompt framework that improves LLM lateral thinking on brain-teaser and riddle QA.
Problem
LLMs struggle with lateral thinking tasks (brain teasers and riddles) that require indirect, creative reasoning and awareness of their own reasoning process.
Approach
Mimics human metacognition in prompting: first formulate a task strategy from cognitive knowledge (person, strategy, task knowledge), then identify implicit wordplay and give an initial answer, then reflect on the reasoning to produce the final answer.
Key results
- With GPT-3.5-turbo, MP beats CoT by +5.00% on Sentence Puzzle, +10.07% on Word Puzzle, +6.48% on BiRdQA, and +2.65% on RiddleSense.
- With GPT-4, MP reaches 98.33% on Sentence Puzzle and 98.96% on Word Puzzle, above prior SOTA (96.67, 97.92) and reported human performance (91.98 on Word Puzzle).
- Ablation: the Reflecting step adds about 3.4 points on Word Puzzle (86.46 to 88.54) and 3.1 on BiRdQA (77.97 to 80.89) for GPT-3.5-turbo.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add metacognitive prompting instead of CoT | GPT-3.5-turbo · BRAINTEASER Sentence Puzzle | accuracy: +5.00% | Helped |
| Add metacognitive prompting instead of CoT | GPT-3.5-turbo · BRAINTEASER Word Puzzle | accuracy: +10.07% | Helped |
| Add metacognitive prompting instead of CoT | GPT-3.5-turbo · BiRdQA | accuracy: +6.48% | Helped |
| Add metacognitive prompting instead of CoT | GPT-3.5-turbo · RiddleSense | accuracy: +2.65% | Helped |
| Add a manually designed Strategy1 to demonstrations (ablation) | MP with GPT-3.5-turbo · BRAINTEASER Sentence Puzzle | accuracy: 64.16 -> 70.00 | Helped |
| Add a manually designed Strategy2 to demonstrations (ablation) | MP with GPT-3.5-turbo · BRAINTEASER Word Puzzle | accuracy: 78.47 -> 83.33 | Helped |
| Add an AI-generated strategy to demonstrations (ablation) | MP with GPT-3.5-turbo · BRAINTEASER Sentence Puzzle | accuracy: 64.16 -> 71.67 | Helped |
| Use a strategy in demonstrations (average of strategy variants) (ablation) | MP with GPT-3.5-turbo · BiRdQA | accuracy: 74.41 -> 77.21 | Helped |
| Add the reflecting step (ablation) | MP with GPT-3.5-turbo · BRAINTEASER Sentence Puzzle | accuracy: 72.50 -> 73.33 | Helped |
Surprising findings
- CoT often underperforms standard prompting and APE with several open models (e.g., Qwen1.5-14B on Sentence Puzzle, 48.33 vs 51.67 STD), suggesting CoT is ill-suited to lateral thinking tasks.
- Human explanation-usefulness evaluation found over 84% of MP explanations let evaluators derive the correct answer.
Limitations
- Error analysis shows the model is misled by distracting information and overthinks options, which can negate correct answers.
- Relies on manually designed strategies and few-shot demonstrations; the paper notes LLMs still lack letter-level and cultural knowledge.
- Evaluation limited to multiple-choice datasets and English riddles only.
Experimental setup
Datasets: BRAINTEASER BiRdQA RiddleSense
Benchmarks: BRAINTEASER Sentence Puzzle BRAINTEASER Word Puzzle BiRdQA RiddleSense
Metrics: accuracy relevance (human) usefulness (human)
Baselines: Standard prompting CoT APE Li et al. 2024 (extended prompt with ensemble voting) Monazzah and Feghhi 2024 (ensemble and debate)
Builds on: chain-of-thought prompting automatic prompt engineer self-consistency
Closest papers by meaning
- Language Models Are Capable of Metacognitive Monitoring and Control of Their Internal Activations (NeurIPS 2025 · similarity 0.87)
- IDEAFix: Evaluation Framework for Creative Defixation Prompting in LLMs (NeurIPS 2026 · similarity 0.86)
- Weak-eval-Strong: Evaluating and Eliciting Lateral Thinking of LLMs with Situation Puzzles (NeurIPS 2024 · similarity 0.86)
- Decoupling Metacognition from Cognition: A Framework for Quantifying Metacognitive Ability in LLMs (AAAI 2025 · similarity 0.86)
- Unveiling the Learning Mind of Language Models: A Cognitive Framework and Empirical Study (NeurIPS 2025 · similarity 0.86)
- CogBench: a large language model walks into a psychology lab (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.