Research map / Large language models / Reasoning and chain of thought

RFEval: Benchmarking Reasoning Faithfulness under Counterfactual Reasoning Intervention in Large Reasoning Models

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Long CoT dynamics and efficiency · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces RFEval, a 7,186-instance benchmark that measures reasoning faithfulness of large reasoning models via output-level counterfactual reasoning interventions, finding 49.7% unfaithful outputs across 12 open LRMs.

Problem

Large reasoning models often produce plausible rationales that do not reflect their actual decision process, and accuracy alone does not reveal this. The paper needs a behavioral way to test whether stated reasoning is internally consistent and causally drives the answer.

Approach

Formalizes reasoning faithfulness as two testable conditions, stance consistency and causal influence under output-level counterfactual reasoning injection, and operationalizes them in a benchmark where flawed reasoning is appended to the model's thinking and the resulting stance change is measured.

Key results

• Across 12 open LRMs, 49.73% of contrast-satisfying instances are unfaithful; overall RF ranges from 27.50% (gpt-oss-120b) to 73.29% (Qwen3-32B).
• Code generation (24.18%) and mathematical reasoning (28.06%) have the lowest RF, while legal decision (70.17%) is highest among tasks.
• Within-family ablations show SFT+RL lowers RF versus SFT-only (MiMo-7B 60.05% to 46.32%; Olmo-3-7B 61.38% to 50.93%).
• Weighted correlation between accuracy and RF after controlling for model and task fixed effects is insignificant (Pearson r=0.090, p≈0.445).

Surprising findings

• Adding RL-style post-training on top of SFT reduces reasoning faithfulness even when accuracy is maintained.
• Average reasoning-step reward from Open-R1 is slightly higher for stance-inconsistent outputs than for consistent ones.

Limitations

• Relies on an LLM evaluator (o3) for stance extraction and flaw identification, which may introduce bias and cannot reveal true internal computation.
• Closed-source models were excluded from main results due to response integrity mechanisms that block trace editing.
• Instantiated at coarse (reasoning, explanation, answer) granularity rather than per-step, and counterfactual reasoning is generated by o3 with limited locality (mean externality penalty 0.395).

Experimental setup

Datasets: LiveCodeBench DS-1000 MMLU GSM8K PrOntoQA RuleBERT-Union-Rules SCITAB PubMedQA PeerRead

Benchmarks: RFEval

Metrics: contrast-conditional reasoning faithfulness (RF) contrast coverage stance extraction micro-F1 flaw identification accuracy Cohen-style agreement (PABAK, percent agreement)

Baselines: Qwen3-8B Qwen3-32B DeepSeek-R1-Distill-Qwen-7B gpt-oss-20b MiMo-7B-RL Llama-3.3-Nemotron-Super-49B

Builds on: LiveCodeBench MMLU GSM8K PrOntoQA SCITAB

Closest papers by meaning

• Measuring the Faithfulness of Thinking Drafts in Large Reasoning Models (NeurIPS 2025 · similarity 0.92)
• FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.90)
• LYNX: Learning Dynamic Exits for Confidence-Controlled Reasoning (NeurIPS 2026 · similarity 0.90)
• Decoding the Critique Mechanism in Large Reasoning Models (NeurIPS 2026 · similarity 0.90)
• Trade-offs in Large Reasoning Models: An Empirical Analysis of Deliberative and Adaptive Reasoning over Foundational Capabilities (AAAI 2026 · similarity 0.89)
• Efficient Reasoning with Balanced Thinking (ICLR 2026 · similarity 0.89)