Research map / Large language models / Reasoning and chain of thought

Monitoring Monitorability

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Long CoT dynamics and efficiency · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes three archetypes (intervention, process, outcome-property) and a g-mean² monitorability metric to evaluate how well CoT monitors detect properties of reasoning-model behavior, then uses the suite to study scaling with reasoning effort, RL, pretraining size, monitor compute, and follow-up questions.

Problem

Chain-of-thought monitorability may degrade under different training procedures, data, or scaling, and there is no broad, trustworthy suite or two-sided metric for measuring it across models and monitors.

Approach

A taxonomy of three monitorability evaluation archetypes plus a g-mean² metric (TPR×TNR) with a minimal-criterion TPR for intervention evals, which handles stochastic sampling and class-imbalance differences across models.

Key results

• Monitorability (g-mean^2) generally increases with CoT length; CoT-only monitors substantially outperform action-only monitors across reasoning efforts.
• Stronger monitors (GPT-5 vs o3-mini) increase g-mean^2 by nearly 40%; scaling o4-mini monitor reasoning effort improves monitorability by ~12%, versus ~4% for action-only.
• RL on frontier runs (o3, GPT-5.1 Codex Max) does not materially decrease monitorability on average, though one run showed a mid-training decrease.

Surprising findings

• Generic CoT obfuscation reward mainly reduced CoT length, and later CoTs degenerated to filler words like 'Weasel' rather than steganography-like encoding.
• Claude 3.7 Sonnet has relatively high action-only monitorability but the lowest CoT monitorability among evaluated models.

Limitations

• Only OpenAI models were tested as monitors, so results may be inflated by shared idiosyncrasies.
• Outcome-property results are often limited by low TNR and ambiguous ground-truth labels in ShadeArena and OAI-repo-sabotage.
• Pretraining scaling experiments use a single seed per scale; the medium model is an outlier.

Experimental setup

Datasets: GPQA AIME Daft Scruples Health Queries (de-identified health data) WMDP FrontierMath Livebench CharXiv ShadeArena Anti-scheming OpenAI repo sabotage codebase

Benchmarks: Health Queries Scruples Memory Toy Math/Science Sandbagging Process Math Process Physics Coding-impossible-tasks Flaky-tools OAI-repo-sabotage ShadeArena Anti-scheming

Metrics: g-mean^2 TPR TNR

Baselines: Claude 3.7 Sonnet (Thinking) DeepSeek R1-0528-Qwen3-8B Kimi K2 Thinking action-only monitoring

Builds on: g-mean F1 metric hint-based faithfulness evaluations

Closest papers by meaning

• CoT Red-Handed: Stress Testing Chain-of-Thought Monitoring (NeurIPS 2025 · similarity 0.91)
• Reasoning Models Struggle to Control their Chains of Thought (ICML 2026 · similarity 0.89)
• Monitorability as a Free Gift: How RLVR Spontaneously Aligns Reasoning (ICML 2026 · similarity 0.88)
• Reasoning Models Sometimes Output Illegible Chains of Thought (NeurIPS 2025 · similarity 0.87)
• Output Supervision Can Obfuscate the Chain of Thought (ICLR 2026 · similarity 0.87)
• Towards a Science of AI Agent Reliability (ICML 2026 · similarity 0.86)