Research map / Large language models / Reasoning and chain of thought

Chain-of-Thought Unfaithfulness as Disguised Accuracy

NeurIPS 2024 · Large language models › Reasoning and chain of thought · approach: Long CoT dynamics and efficiency

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Replicates Lanham et al.'s CoT-unfaithfulness scaling study on Llama 2, FLAN-T5/UL2, and Pythia DPO, and shows that normalizing for answer-letter bias removes most of the apparent inverse scaling and that the metric correlates strongly with accuracy.

Problem

Lanham et al. report that CoT unfaithfulness (answer change with vs. without CoT) follows a V-shaped scaling with model size within one proprietary family, peaking near 13B. It is unclear whether this holds for LLMs generally, and whether the metric is confounded.

Approach

Replicate the scaling experiments on three open model families and introduce a normalized unfaithfulness metric that divides by the rate at which the model picks the same letter under answer-choice shuffling in the No-CoT setting, showing the apparent unfaithfulness of weak models is largely letter bias and that the metric tracks accuracy.

Key results

• Normalized unfaithfulness shows a much shallower V-shape for Pythia DPO than the unnormalized metric, and Llama 2 unfaithfulness still shows inverse scaling after normalization.
• Normalized unfaithfulness correlates strongly with accuracy (R^2=0.740), while unnormalized unfaithfulness shows minimal correlation (R^2=0.025).
• Llama 2 shows the most faithful CoTs at 13B on the addition task for both 2- and 3-digit problems, contrary to the expectation that difficulty shifts the optimum.

Measured effects reported in the paper

Surprising findings

• Normalized unfaithfulness is strongly linearly correlated with accuracy (R^2=0.74), suggesting the metric may mostly measure task capability.
• Pythia DPO 70M shows near-100% unfaithfulness that largely disappears after accounting for answer-choice order sensitivity.

Limitations

• Only three model families, with gaps around the 13B point for FLAN-T5 and Pythia DPO.
• Subsampled evaluation sets (up to 500 examples) and sparse model sizes for Llama 2 limit granularity.
• Could not evaluate the original proprietary Anthropic models; relied on quantized 4-bit/8-bit variants for the largest models.

Experimental setup

Datasets: AQuA-RAT ARC-Challenge ARC-Easy HellaSwag LogiQA MMLU OpenBookQA TruthfulQA

Benchmarks: AQuA-RAT ARC-Challenge ARC-Easy HellaSwag LogiQA MMLU OpenBookQA TruthfulQA 2-digit and 3-digit addition

Metrics: unfaithfulness (Lanham) normalized unfaithfulness accuracy R^2 correlation

Baselines: Lanham et al. unfaithfulness metric

Builds on: Lanham et al. CoT faithfulness tests direct preference optimization Pythia

Closest papers by meaning

• FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.85)
• Chain-of-Thought Reasoning In The Wild Is Not Always Faithful (ICML 2026 · similarity 0.85)
• Reliable Chain-of-Thought via Prefix Consistency (NeurIPS 2026 · similarity 0.83)
• Bounds of Chain-of-Thought Robustness: Reasoning Steps, Embed Norms, and Beyond (ICLR 2026 · similarity 0.83)
• BadThink: Triggered Overthinking Attacks on Chain-of-Thought Reasoning in Large Language Models (AAAI 2026 · similarity 0.82)
• Deep Hidden Cognition Facilitates Reliable Chain-of-Thought Reasoning (AAAI 2026 · similarity 0.82)