Research map / Large language models / Reasoning and chain of thought

LINGOLY-TOO: Disentangling Reasoning from Knowledge with Templatised Orthographic Obfuscation

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Long CoT dynamics and efficiency · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces LingOLY-TOO, a 1,203-question (6,995 sub-question) linguistics-olympiad benchmark whose expert-designed grapheme permutations remove knowledge/memorisation shortcuts, showing frontier model scores drop from ~0.59 to ~0.48 under obfuscation.

Problem

Frontier LLMs score well on reasoning benchmarks partly through memorisation and stored linguistic knowledge rather than genuine reasoning, inflating estimates of reasoning ability.

Approach

Expert-designed, linguistically-informed grapheme-level permutations (reasoning-equivariant obfuscation) of Linguistics Olympiad problems preserve solution logic while making original-language knowledge useless, and the gap between original and obfuscated scores quantifies knowledge-driven shortcuts.

Key results

• Best frontier models score about 0.59 on original problems but drop to about 0.48 under obfuscation; GPT-5 scores 48% overall and 31% on Round 2 problems.
• Under the robust metric M_rob, GPT-5 achieves only 0.29, a decrease of over 0.19.
• In a 172-subject human study, human performance dropped 5.70% on obfuscated problems, compared with 5.8% for reasoning models.

Surprising findings

• Language resourcedness correlates negatively with the original-vs-obfuscated gap at higher difficulty levels, suggesting models guess better in high-resource languages.
• Tokenisation changes (dashes or per-character tokenisation) did not improve performance on original or obfuscated problems.

Limitations

• Exact match penalises partially correct answers and gives limited insight into failure modes.
• Scope limited to inductive and deductive reasoning in natural language; not a comprehensive reasoning evaluation.
• Obfuscation may impose a cognitive penalty for humans familiar with original orthographies.

Experimental setup

Datasets: UKLO Linguistics Olympiad problems

Benchmarks: LingOLY-TOO LingOLY

Metrics: exact match obfuscated score (M_obf) original score (M_og) robust score (M_rob)

Baselines: GPT-5 GPT-4.5 GPT-4o Claude 3.7 Sonnet o3-mini Llama 3.3 70B Gemini 1.5 Pro Claude 3.5

Builds on: LingOly UKLO

Closest papers by meaning

• LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low Resource and Extinct Languages (NeurIPS 2024 · similarity 0.90)
• Linguini: A benchmark for language-agnostic linguistic reasoning (NeurIPS 2025 · similarity 0.89)
• Are Language Models Efficient Reasoners? A Perspective from Logic Programming (NeurIPS 2025 · similarity 0.88)
• Beyond Semantics: The Unreasonable Effectiveness of Reasonless Intermediate Tokens (NeurIPS 2026 · similarity 0.88)
• The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity (NeurIPS 2025 · similarity 0.88)
• Mitigating Overthinking in Large Reasoning Language Models via Reasoning Path Deviation Monitoring (NeurIPS 2026 · similarity 0.87)