Research map / Large language models / Agents, code and math
On the Eligibility of LLMs for Counterfactual Reasoning: A Decompositional Study
Shuai Yang, Qi Yang, Luoxi Tang, Yuqiao Meng, Nancy Guo, Jeremy Blackburn, Zhaohan Xi
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use harder modality (code) instead of text | LLMs on causal variable identification · Task I | F1: CRASS 92.3 -> Open-Critic 71.7 (GPT-5, X) | Hurt |
| identify implicit mediator M instead of explicit variables | LLMs · Task I (text) | F1: CRASS GPT-5: X 92.3 vs M 87.4 | Hurt |
| construct causal graph given ground-truth variables | LLMs · Task II | F1 | No clear effect |
| reason about counterfactual M' and Y' (implicit) vs factual | LLMs · Task IV | F1: CRASS GPT-5: M' 92.1, Y' 88.0; Open-Critic: 75.3, 69.4 | Hurt |
| add tool-augmented entity extraction (NER, grounding-dino, GraphCodeBERT) (ablation) | GPT-o4, Qwen3, Llama4-S, Gemini2.5 · explicit variables X, Z, Y | F1: up to +32.0 F1 points (CVQA-Count, Llama4-S X) | Improved |
| add CoT prompting (ablation) | GPT-o4, Qwen3, Llama4-S, Gemini2.5 · implicit variables M, M', Y' | F1: +5.6 to +7.9 on CRASS M | Improved |
| use CoT-SC or ToT instead of CoT (ablation) | LLMs · implicit variables M, M', Y' | F1: CLOMO GPT-o4 M': CoT +4.9, ToT +3.8 | Mixed |
| combine tool augmentation and elicitation strategies (ablation) | GPT-5, Qwen3, Llama4-S, Gemini2.5 · end-to-end Y' | F1: Open-Critic Llama4-S: +2.7 / +11.2 / +14.2 | Improved |
| improve implicit variable reasoning versus explicit variable identification (ablation) | LLMs · end-to-end Y' | F1: CRASS GPT-5: +1.8 vs +6.2 | Improved |
Closest papers by meaning
- What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models (CVPR 2024 · similarity 0.92)
- Reasoning Elicitation in Language Models via Counterfactual Feedback (ICLR 2025 · similarity 0.91)
- Counterfactual reasoning: an analysis of in-context emergence (NeurIPS 2025 · similarity 0.91)
- Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanations (ICML 2024 · similarity 0.90)
- Abstract Counterfactuals for Language Model Agents (NeurIPS 2025 · similarity 0.89)
- Does Reasoning Emerge? Examining the Probabilities of Causation in Large Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
