Research map / Large language models / Retrieval and knowledge
Linguini: A benchmark for language-agnostic linguistic reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Linguini, a benchmark of 894 linguistics olympiad questions (160 problems, 75 mostly low-resource languages) to test language-agnostic linguistic reasoning, finding all models below 25% exact match.
Problem
Existing language evaluation benchmarks conflate linguistic reasoning with language-specific pre-trained knowledge and are saturating, so it is hard to measure meta-linguistic deductive skills on languages models have not seen.
Approach
Builds a benchmark from International Linguistic Olympiad Rosetta Stone-style problems on mostly extremely low-resource languages, where all information needed is in the context, and uses ablations (no-context, script substitution, textbook-in-context) to check contamination.
Key results
- Best proprietary model claude-3-opus reaches 24.05% exact match zero-shot; best open model llama-3-70b reaches 8.84% with 3 in-context examples.
- No-context prompting drops claude-3-opus from 24.05% to 1.23%, supporting reliance on in-context information rather than memorized language data.
- Claude-3-opus solves 13 of 16 transliterated problems at least as well in some non-Latin script as in Latin script.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Remove the provided linguistic context (ablation) | Linguini model evaluations · Linguini | accuracy: -3.69 to -22.82 | Hurt |
| Increase in-context examples from zero to five (ablation) | claude-3-opus · Linguini | accuracy: 24.05 -> 15.1 | Hurt |
| Increase in-context examples from zero to five (ablation) | llama-3-70b-it · Linguini | accuracy: 4.81 -> 8.39 | Helped |
| Increase in-context examples from zero to five (ablation) | gpt-4 · Linguini | accuracy: 6.38 -> 13.31 | Helped |
| Transliterate Latin script into non-Latin scripts (ablation) | claude-3-opus · Selected 16 problems | accuracy: 85.71; 56.12; 65.31; 63.27; 38.78 | Mixed |
| Remove the benchmark context (ablation) | models in the One-Book subset · One-Book subset (average) | accuracy: 1.71 -> 0.00 | Hurt |
| Add a textbook alongside the benchmark context (ablation) | models in the One-Book subset · One-Book subset (average) | accuracy: 1.71 -> 6.84 | Helped |
| Use a textbook without the benchmark context (ablation) | models in the One-Book subset · One-Book subset (average) | accuracy: 0.00 -> 0.00 | No clear effect |
| Use five in-context examples instead of zero (ablation) | gpt-4-turbo · Linguini | accuracy: 8.72 -> 9.96 | Mixed |
| Use five in-context examples instead of zero (ablation) | claude-3-sonnet · Linguini | accuracy: 12.30 -> 8.72 | Hurt |
| Use five in-context examples instead of zero (ablation) | llama-3-70b · Linguini | accuracy: 8.17 -> 6.60 | Mixed |
| Use five in-context examples instead of zero (ablation) | llama-2-70b-it · Linguini | accuracy: 0.89 -> 2.80 | Helped |
Surprising findings
- Performance of several models degrades as the number of in-context examples increases (e.g., claude-3-opus from 24.05% to 15.1%).
- Accuracy shows a roughly uniform trend against language speaker counts and online search presence, showing little correlation with language resourcefulness.
Limitations
- Long in-context grammar book experiments are limited to a few languages due to availability of grammars.
- No curated explanations per problem, so chain-of-thought experiments are not possible.
- Benchmark is small (160 problems) and results are not reported with error bars.
Experimental setup
Datasets: International Linguistic Olympiad problems (2003-2023)
Benchmarks: Linguini
Metrics: exact match accuracy chrF
Baselines: claude-3-opus gpt-4o gpt-4 claude-3-sonnet gpt-4-turbo llama-3-70b claude-3-haiku llama-2-70b mistral-0.1-8x7b gemma-2b qwen-1.5-110b-it
Builds on: Rosetta Stone-style problems PuzzLing Machines Big-Bench linguistic mappings
Closest papers by meaning
- LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low Resource and Extinct Languages (NeurIPS 2024 · similarity 0.92)
- metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.89)
- Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought (ICLR 2026 · similarity 0.89)
- LINGOLY-TOO: Disentangling Reasoning from Knowledge with Templatised Orthographic Obfuscation (ICLR 2026 · similarity 0.89)
- MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation (ICLR 2025 · similarity 0.89)
- AdaMCoT: Rethinking Cross-Lingual Factual Reasoning Through Adaptive Multilingual Chain-of-Thought (AAAI 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.