Research map / Large language models / Agents, code and math
A Reasoning-Based Approach to Cryptic Crossword Clue Solving
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds an open-licensed LLM pipeline for cryptic crossword clues that hypothesises answers, proposes wordplay, and verifies Python formalisations of the wordplay, reaching state-of-the-art Top-1 accuracy on Cryptonite.
Problem
Solving cryptic crossword clues requires multi-step wordplay reasoning that LLMs handle poorly; prior LLM and rule-based solvers scored very low on the Cryptonite benchmark.
Approach
Generate answer candidates first, then wordplay hypotheses per answer, then formalise each wordplay as Python assert-based proofs using in-context prompting, and check them with an AST-based line-by-line verifier that feeds hints back for rewrites.
Key results
- Gemini-Flash formaliser pipeline reaches 32.5% Top-1 accuracy on the Cryptonite test set, versus 26.0%-ish range for GPT-4o 5-shot at 27.6%, claimed as new SOTA.
- Gemma2-9B-it formaliser (no fine-tuning of formaliser) reaches 29.0% test accuracy, exceeding the GPT-4o 5-shot baseline of 27.6% on the sampled test set.
- Gemma2-9B frequency voting over 20 answer candidates reaches 25.5% test accuracy, vs 6.5% for Gemini-Flash 5-shot.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune Gemma2-9B on Cryptonite clue-answer examples | Gemma2-9B answer generation · Cryptonite test | Top-1 accuracy: 4.5% -> 15.9% | Helped |
| Choose the most frequent answer among 20 generated candidates instead of the first answer | Gemma2-9B answer generation · Cryptonite test | Top-1 accuracy: 15.9% -> 25.5% | Helped |
| Use the full candidate, wordplay, formalisation, and verification pipeline rather than frequency-based answer selection | Gemini-Flash Formaliser · Cryptonite test | Top-1 accuracy: 25.5% -> 32.5% | Helped |
| Replace Gemini-Flash with Gemma2-9B-it for proof formalisation | Cryptic crossword reasoning pipeline · Cryptonite test | Top-1 accuracy: 32.5% -> 29.0% | Hurt |
| Choose candidate answers by highest answer log probability instead of using the full verification pipeline (ablation) | Gemma2-9B reasoning pipeline · Cryptonite test | Top-1 accuracy: 25.5% -> 22.7% | Hurt |
| Choose answers by the wordplay model's highest log probability instead of using the full verification pipeline (ablation) | Gemma2-9B reasoning pipeline · Cryptonite test | Top-1 accuracy: 25.5% -> 20.5% | Hurt |
Surprising findings
- Ranking wordplay by model logprob performed worse than verification, indicating fictitious but simple wordplay gets high likelihood.
- Gemini-Flash 5-shot and Gemma2-9B-it 5-shot score very poorly (6.5% and 4.5% test), far below GPT-4o.
Limitations
- Answer candidate generation caps the upper bound; the correct answer appears in top-20 only about 45% of the time.
- Verifier has shortcuts (comments-only functions, conditional routing, assert==False rewrites), so reward hacking is possible if used with RL.
- Sampled evaluation on about 1000 validation/test clues and 200 for formaliser gives roughly +/-1.5% to +/-3.3% standard deviation.
Experimental setup
Datasets: Cryptonite Wordplay dataset crossword words list UK Advanced Cryptics Dictionary
Benchmarks: Cryptonite
Metrics: Top-1 exact-match accuracy Bayesian IRT probability of outperforming GPT-4o
Baselines: GPT-4o 5-shot Gemini-Flash 5-shot Gemma2-9B-it 5-shot Rule-based solver T5-large fine-tuned Gemma2-9B FT
Builds on: PAL Draft, Sketch, and Prove AlphaCodium Self-Debug tool-integrated reasoning
Closest papers by meaning
- Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzles (NeurIPS 2025 · similarity 0.83)
- LINGOLY-TOO: Disentangling Reasoning from Knowledge with Templatised Orthographic Obfuscation (ICLR 2026 · similarity 0.82)
- Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation (ICML 2025 · similarity 0.81)
- ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning (ICML 2025 · similarity 0.81)
- LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low Resource and Extinct Languages (NeurIPS 2024 · similarity 0.81)
- Logic.py: Bridging the Gap between LLMs and Constraint Solvers (NeurIPS 2025 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.