Research map / Large language models / Agents, code and math
Weak-eval-Strong: Evaluating and Eliciting Lateral Thinking of LLMs with Situation Puzzles
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces SPLAT, a 975-puzzle situation-puzzle benchmark for lateral thinking, scored with a multi-turn player-judge game where a judge model holding the reference scenario answers yes/no questions and checks final deductions.
Problem
Evaluating lateral thinking of LLMs is hard because creative open-ended answers are difficult to judge, and model-based evaluation usually requires a judge stronger than the models being tested.
Approach
Replace static model-based grading with an interactive player-judge game on situation puzzles: the tested model (player) asks yes/no/irrelevant questions about an incomplete story, and a judge that alone sees the reference scenario answers and semantically checks the final deduction, so the judge need not be stronger than the player.
Key results
- WizardLM-2 as judge agrees with humans at 100%, 87.50%, 88.24% on final-answer judgements across Easy, Medium, Hard puzzles.
- GPT-4 achieves the highest average accuracy on SPLAT at 41.70% with 11.77 average rounds; Llama3-70B reaches 34.50%.
- Adding SPLAT-derived question-answer auxiliary prompts raises Llama3-70B on RiddleSense dev from 83.34% to 85.21%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add SPLAT auxiliary reasoning prompts | Llama3-8B · RiddleSense dev | accuracy: 70.51% -> 72.18% | Helped |
| Add SPLAT auxiliary reasoning prompts | Llama3-70B · RiddleSense dev | accuracy: 83.34% -> 85.21% | Helped |
| Add SPLAT auxiliary reasoning prompts | GPT-4 · RiddleSense dev | accuracy: 88.83% -> 89.23% | Helped |
| Add SPLAT data only (ablation) | Llama3-8B · RiddleSense dev | accuracy: 70.51 -> 70.32 | Hurt |
| Add SPLAT data only (ablation) | Llama3-70B · RiddleSense dev | accuracy: 83.34 -> 82.95 | Hurt |
| Add SPLAT data only (ablation) | Llama3-8B · BrainTeaser Sentence | overall accuracy: 67.65 -> 69.62 | Helped |
| Add SPLAT data only (ablation) | Llama3-8B · BrainTeaser Word | overall accuracy: 46.20 -> 52.28 | Helped |
| Add SPLAT data only (ablation) | Llama3-70B · BrainTeaser Sentence | overall accuracy: 87.76 -> 91.12 | Helped |
| Add SPLAT data only (ablation) | Llama3-70B · BrainTeaser Word | overall accuracy: 71.20 -> 77.27 | Helped |
| Add SPLAT reasoning processes in addition to SPLAT data (ablation) | Llama3-8B · Average accuracy across RiddleSense and BrainTeaser | average accuracy: 64.07 -> 66.71 | Helped |
| Add SPLAT reasoning processes in addition to SPLAT data (ablation) | Llama3-70B · Average accuracy across RiddleSense and BrainTeaser | average accuracy: 83.78 -> 85.42 | Helped |
| Use WizardLM-2 instead of Llama3-70B as the judge | SPLAT judge evaluation · SPLAT Hard reasoning process | agreement with humans: 71.36% -> 90.15% | Helped |
Surprising findings
- Llama3-70B as judge agrees with humans only 71-84% on reasoning-process judgements, far below WizardLM-2 on hard puzzles (72.14% vs 90.15% with humans for WizardLM-2).
- Qwen1.5-110B-Chat scores lower than much smaller or comparable models, which the authors attribute to chat-configuration issues.
Limitations
- Judge reliability is validated on only 60 puzzles (156 final-answer and 2,271 reasoning judgements), a small human-agreement sample.
- Multiple valid scenarios may exist, so open-ended deductions are hard to score; the repeated-run R parameter is defined but set to 1 in experiments.
- Judge still depends on a strong model (WizardLM-2 8x22B), and the paper notes the judge is not fully free of self-preference concerns.
Experimental setup
Datasets: SPLAT (975 situation puzzles, crawled from public websites) RiddleSense BrainTeaser CSQA
Benchmarks: SPLAT RiddleSense (dev) BrainTeaser
Metrics: Accuracy Average Round OverAll (O/A) agreement with humans
Baselines: Llama3-8B Llama3-70B Qwen1.5-32B Qwen1.5-110B WizardLM-2 GPT-4 Turbo GPT-4
Builds on: GAIA MT-Bench BrainTeaser RiddleSense
Closest papers by meaning
- GameArena: Evaluating LLM Reasoning through Live Computer Games (ICLR 2025 · similarity 0.88)
- Let's Think Outside the Box: Exploring Leap-of-Thought in Large Language Models with Creative Humor Generation (CVPR 2024 · similarity 0.88)
- Large Language Models Play StarCraft II:Benchmarks and A Chain of Summarization Approach (NeurIPS 2024 · similarity 0.87)
- Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models (ICLR 2025 · similarity 0.86)
- ToMATO: Verbalizing the Mental States of Role-Playing LLMs for Benchmarking Theory of Mind (AAAI 2025 · similarity 0.86)
- MP: Endowing Large Language Models with Lateral Thinking (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.