Research map / Large language models / Agents, code and math
LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low Resource and Extinct Languages
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces LingOly, a 1,133-question benchmark from UK Linguistics Olympiad puzzles on low-resource and extinct languages, scored by exact match and a no-context baseline delta to control memorisation.
Problem
Measuring genuine multi-step reasoning in LLMs without contamination or memorisation, using linguistic puzzles in very low-resource and extinct languages.
Approach
Uses UK Linguistics Olympiad puzzles (translation and pattern deduction in low-resource languages) as a reasoning benchmark and adds a no-context baseline (Delta_NC) that penalises answers obtainable from memorisation rather than the provided context.
Key results
- Top model Claude Opus reaches 46.3% exact match and 28.8% Delta_NC overall; average exact match across 12 models is 20.8%.
- On harder problems the top model reaches 38.7% accuracy, a 24.7% improvement over the no-context baseline.
- Closed models outperform open models on both metrics; top open model Mixtral 8x7B achieves 14.2% exact match.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Claude Opus instead of Mixtral 8x7B | Models evaluated on LingOly · LingOly | exact match accuracy: 14.2% -> 46.3% | Helped |
| Use Claude Opus instead of Mixtral 8x7B | Models evaluated on LingOly · LingOly | ΔNC: 6.4% -> 28.8% | Helped |
| Evaluate on higher human-difficulty questions rather than easier ones | LLMs evaluated on LingOly · LingOly difficulty levels | exact match accuracy | Hurt |
| Evaluate on higher-resource rather than lower-resource languages | Models evaluated on LingOly · LingOly languages | exact match accuracy | Helped |
| Use no-context prompts instead of full prompts | LLMs evaluated on LingOly · LingOly | exact match accuracy | Hurt |
Surprising findings
- Models score higher on exact match for higher-resource languages, but the Delta_NC relationship is weaker and not significant for open models.
- Foundation-level questions show a large drop from exact match to Delta_NC, indicating substantial memorisation on those items.
Limitations
- Exact match scoring is harsh on partially correct answers and cannot award partial credit.
- Memorisation and future contamination cannot be fully ruled out, and parsing errors may remain.
- Puzzles requiring images or non-Latin scripts were excluded, so the benchmark is text-only.
Experimental setup
Datasets: UK Linguistics Olympiad puzzles
Benchmarks: LingOly
Metrics: exact match Delta_NC (no-context improvement)
Baselines: Llama 3 70B Llama 3 8B Mixtral 8x7B Aya 23 35B Gemma 7B Llama 2 70B GPT-4o GPT-4 GPT-3.5 Claude Opus Gemini 1.5 Pro Command R+
Builds on: Rosetta Stone translation benchmarks Datasheets for Datasets
Closest papers by meaning
- Linguini: A benchmark for language-agnostic linguistic reasoning (NeurIPS 2025 · similarity 0.92)
- LINGOLY-TOO: Disentangling Reasoning from Knowledge with Templatised Orthographic Obfuscation (ICLR 2026 · similarity 0.90)
- Reasoning Transfer for an Extremely Low-Resource and Endangered Language: Bridging Languages Through Sample-Efficient Language Understanding (AAAI 2026 · similarity 0.89)
- Enigmata: Scaling Logical Reasoning in Large Language Models with Synthetic Verifiable Puzzles (NeurIPS 2025 · similarity 0.89)
- PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts (NeurIPS 2025 · similarity 0.89)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.