Research map / Large language models / Reasoning and chain of thought

All Code, No Thought: Language Models Struggle to Reason in Ciphered Language

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Long CoT dynamics and efficiency

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Tests whether LLMs can reason in 28 ciphered text formats by fine-tuning or few-shot prompting them, finding math accuracy drops sharply for many ciphers even though models can translate them accurately.

Problem

Whether language models can perform reasoning in ciphered, encrypted, translated, or compressed text, which bears on whether CoT monitoring can be evaded via ciphered reasoning.

Approach

Measures ciphered reasoning as math accuracy relative to plain-text reasoning (with cipher adherence), contrasted with cipher translation ability, across 28 ciphers; links capability to cipher prevalence in pretraining data and characterizes data scaling laws.

Key results

• Fine-tuned GPT 4.1 loses 19%+ accuracy on rot13 and up to 72% on reverse fibonacci indices relative to identity.
• Few-shot Sonnet 3.5/4 lose under 10% on rot13 and base64 but over 45% on reverse fibonacci indices.
• Pretraining prevalence of a cipher predicts math accuracy with R^2 of 0.906 for structure-preserving ciphers under per-cluster logistic fits.

Surprising findings

• Models translate ciphered text to fluent English (BLEU above 50) even when they cannot reason accurately in that cipher.
• Few-shot prompted models sometimes game the task by ciphering prose while keeping math expressions in plain text.

Limitations

• Does not design ciphers that intentionally minimize legibility while maximizing reasoning.
• Does not use RL extensively; RL experiments are limited to 4 ciphers on Qwen2.5 7B.
• Cannot fine-tune GPT 5 or Claude models and could not evaluate Claude Opus 4 due to safety classifiers.

Experimental setup

Datasets: NuminaMath-CoT MATH MATH 500 PRM800K RedPajama

Benchmarks: MATH 500 PRM800K cipher translation

Metrics: accuracy relative to identity adherence rate BLEU

Baselines: identity (plain-text reasoning) direct answering generally prompted translation

Builds on: GRPO Infini-gram NuminaMath-CoT

Closest papers by meaning

• Large language models can learn and generalize steganographic chain-of-thought under process supervision (NeurIPS 2025 · similarity 0.88)
• Reasoning Models Sometimes Output Illegible Chains of Thought (NeurIPS 2025 · similarity 0.87)
• BadThink: Triggered Overthinking Attacks on Chain-of-Thought Reasoning in Large Language Models (AAAI 2026 · similarity 0.87)
• CoT-Guard: Small Models for Strong Monitoring (NeurIPS 2026 · similarity 0.86)
• CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring (NeurIPS 2025 · similarity 0.86)
• Let Models Speak Ciphers: Multiagent Debate through Embeddings (ICLR 2024 · similarity 0.86)