Research map / Large language models / Reasoning and chain of thought

When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMs

NeurIPS 2025 · Large language models › Reasoning and chain of thought · approach: Chain-of-thought training and analysis · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that chain-of-thought prompting degrades instruction-following accuracy on IFEval and ComplexBench across many LLMs, analyzes causes via case studies and constraint attention, and evaluates four mitigation strategies.

Problem

Whether explicit reasoning (chain-of-thought, or reasoning-tuned models) helps or hurts instruction-following accuracy under rule-verifiable and compositional constraints is unclear.

Approach

Empirically finds CoT often lowers instruction-following scores, attributes this partly to reduced attention on constraint tokens (a "constraint attention" metric), and proposes mitigations including few-shot ICL, self-reflection, self-selective reasoning, and a per-model classifier that chooses whether to use CoT.

Key results

• 13 of 14 models lose accuracy on IFEval with CoT; all models decline on ComplexBench (e.g., Llama3-8B-Instruct 75.2% to 59.0% on IFEval).
• Classifier-selective reasoning improves Meta-Llama-3-8B-Instruct on IFEval from 59.0% (CoT) to 69.7%, and self-reflection reaches 87.6% for Claude-3.5-Sonnet vs 79.5% CoT.
• Larger attention drops on constraint tokens are seen in LOSE cases than WIN cases, particularly in early-to-middle layers.

Measured effects reported in the paper

Surprising findings

• Reasoning length does not meaningfully correlate with instruction-following effectiveness.
• Few-shot in-context learning with corrected reasoning examples often hurts weaker models such as Llama-3.2-1B-Instruct on IFEval (49.0 original, 13.1 few-shot).

Limitations

• Only studies instruction-following; generalization to other domains is untested.
• Classifier-selective reasoning requires a separate trained classifier per target model.
• Comparisons between reasoning-tuned and base models are not fully controlled.

Experimental setup

Datasets: IFEval ComplexBench

Benchmarks: IFEval ComplexBench

Metrics: loose instruction-level accuracy proportion of constraints satisfied constraint attention attention drop

Baselines: CoT prompting few-shot in-context learning self-reflection self-selective reasoning

Builds on: chain-of-thought prompting IFEval ComplexBench

Closest papers by meaning

• Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models (NeurIPS 2025 · similarity 0.93)
• Light-IF: Endowing LLMs with Generalizable Reasoning via Preview and Self-Checking for Complex Instruction Following (AAAI 2026 · similarity 0.91)
• Facts Don't Speak Louder than Words: The Behavioral Essence of Long-CoT Distillation (NeurIPS 2026 · similarity 0.91)
• SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought (NeurIPS 2025 · similarity 0.91)
• CoT Vectors: Transferring and Probing the Reasoning Mechanisms of LLMs (ICLR 2026 · similarity 0.90)
• Large Language Models as Analogical Reasoners (ICLR 2024 · similarity 0.90)