Research map / Large language models / Reasoning and chain of thought

Reasoning Models Sometimes Output Illegible Chains of Thought

NeurIPS 2025 · Large language models › Reasoning and chain of thought · approach: Long CoT dynamics and efficiency · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Evaluates chain-of-thought legibility across 14 reasoning and non-reasoning models on GPQA-Diamond questions, finding RL-trained models often produce illegible CoTs (except Claude) that still contribute to accuracy.

Problem

Whether outcome-based RL makes reasoning models' chain-of-thought illegible to humans and AI monitors, which would undermine CoT monitoring for safety.

Approach

Measures CoT legibility with a GPT-4o autograder (1-9 scale) across many reasoning models, then uses prefill truncation and per-question resampling to test whether illegible segments are functionally useful, and correlates illegibility with question difficulty.

Key results

• Every reasoning model trained with outcome-based RL except Claude and Kimi shows meaningful illegibility; R1 produces CoT chunks scoring >=7 (entirely inscrutable) in 29.4% of cases.
• Forcing QwQ to answer from only legible CoT portions raises fully-wrong answers from 54.1% to 73.6% and drops definitely-correct answers from 24.6% to 11.5% (a 53% decrease).
• Per-question correlation between normalized illegibility and correctness for QwQ averages near zero (mean r = 0.061).

Measured effects reported in the paper

Surprising findings

• Across resampled QwQ answers, legibility shows no consistent correlation with correctness despite prefill truncation hurting accuracy.
• Larger reasoning models in the same paradigm tend to produce more illegible CoTs, suggesting illegibility is not purely a capability limitation.

Limitations

• Causal mechanisms behind illegible CoTs cannot be determined from observations alone.
• Evaluation limited to GPQA-Diamond single-turn science QA, which may not generalize to other monitoring tasks.
• OpenAI reasoning model CoTs are not publicly exposed, so frontier comparisons rely on individual samples.

Experimental setup

Datasets: GPQA-Diamond

Benchmarks: GPQA-Diamond

Metrics: legibility score (1-9) accuracy Cohen's weighted kappa Spearman correlation per-question Pearson correlation

Baselines: DeepSeek V3 Claude Sonnet 4 Claude Opus 4 Qwen3-235B R1-Distill-Qwen-32B R1-Distill-Qwen-14B

Builds on: GPQA-Diamond R1-Zero chain-of-thought faithfulness evaluation

Closest papers by meaning

• Reasoning Models Struggle to Control their Chains of Thought (ICML 2026 · similarity 0.90)
• FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.90)
• How Does Chain of Thought Think? Mechanistic Interpretability of Chain-of-Thought Reasoning with Sparse Autoencoding (AAAI 2026 · similarity 0.89)
• Facts Don't Speak Louder than Words: The Behavioral Essence of Long-CoT Distillation (NeurIPS 2026 · similarity 0.89)
• When More is Less: Understanding Chain-of-Thought Length in LLMs (ICLR 2026 · similarity 0.89)
• Deep Hidden Cognition Facilitates Reliable Chain-of-Thought Reasoning (AAAI 2026 · similarity 0.89)