Research map / Large language models / Reasoning and chain of thought
LLMs Must Think Thrice to Solve Executable Counterfactuals
Aniket Vashishtha, Qirun Dai, Hongyuan Mei, Amit Sharma, Chenhao Tan, Hao Peng
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies LLM counterfactual reasoning through executable programs covering abduction, intervention, and prediction steps; the abstract is truncated so the proposed method is not stated.
Problem
LLMs need counterfactual reasoning (abduction, intervention, prediction) for causal understanding in high-stakes domains such as science and healthcare; the abstract is truncated, so the specific gap is only partly known.
Approach
Not stated in the provided abstract (truncated). The title suggests that LLMs must reason across the three counterfactual steps, i.e., think thrice, to solve executable counterfactuals.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train with RLVR (GRPO) on counterfactual code | Qwen2.5-7B-Instruct · if_else (ID) | exact match: 13.9 -> 67.8 | Improved |
| train with RLVR (GRPO) | Qwen2.5-7B-Instruct · while (OOD) | exact match: 3.3 -> 8.1 | Improved |
| train with RLVR (GRPO) | Qwen2.5-7B-Instruct · multi_r (OOD) | exact match: 17.9 -> 36.3 | Improved |
| train with RLVR (GRPO) | Qwen2.5-7B-Instruct · if_else-long (OOD) | exact match: 28.2 -> 58.3 | Improved |
| distillation SFT on R1-Distill-Qwen-32B traces | Qwen2.5-7B-Instruct · if_else (ID) | exact match: 13.9 -> 59.0 | Improved |
| distillation SFT on R1-Distill-Qwen-32B traces | Qwen2.5-7B-Instruct · while (OOD) | exact match: 3.3 -> 2.1 | Hurt |
| distillation SFT on R1-Distill-Qwen-32B traces | Qwen2.5-7B-Instruct · multi_r (OOD) | exact match: 17.9 -> 23.3 | Mixed |
| distillation SFT on code traces | Qwen2.5-7B-Instruct · GSM counterfactual (code-to-math) | accuracy: 39.0 -> 12.9 | Hurt |
| RLVR on code traces | Qwen2.5-7B-Instruct · GSM counterfactual (code-to-math) | accuracy: 39.0 -> 46.3 | Improved |
| reveal the latent variable (interventional instead of counterfactual) | Qwen2.5-7B-Instruct · GSM | accuracy: 39.0 -> 60.4 | Improved |
| scale model size from 7B to 72B | Qwen2.5-Instruct · while counterfactual | exact match: 3.3 -> 9.0 | Mixed |
Limitations
- Abstract is truncated; full method and results unavailable
Closest papers by meaning
- What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models (CVPR 2024 · similarity 0.89)
- Counterfactual reasoning: an analysis of in-context emergence (NeurIPS 2025 · similarity 0.88)
- One Example Shown, Many Concepts Known! Counterexample-Driven Conceptual Reasoning in Mathematical LLMs (ICML 2025 · similarity 0.88)
- RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Fold (NeurIPS 2024 · similarity 0.88)
- Reducing Credit Assignment Variance via Counterfactual Reasoning Paths (NeurIPS 2026 · similarity 0.88)
- Counterfactual Identification Under Monotonicity Constraints (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
