Research map / Large language models / Language, safety and interpretability

Eliciting Causal Abilities in Large Language Models for Reasoning Tasks

AAAI 2025 · Large language models › Language, safety and interpretability · approach: Faithfulness of LLM explanations · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Self-Causal Instruction Enhancement (SCIE), where an LLM generates observational data on instruction proxy features, estimates ATE via T-Learner-style in-context code execution, and rewrites instructions to raise the overall ATE, plus an Object-Relational module for reusing learned causal classes across tasks.

Problem

Prompt optimization methods for LLM reasoning are costly to train (gradient-based) or lack interpretability (edit/search-based); the paper asks whether causal inference can be elicited from LLMs to improve instructions cheaply and interpretably.

Approach

Treat instructions as treatments and answer correctness as outcome; have the LLM generate counterfactual instruction variants along LLM-scored textual proxy features, estimate ATE per feature (T-Learner via in-context code execution), and ask the LLM to write new instructions that increase the overall ATE; reuse the learned causal relations as an inheritable class across tasks via object-relationa

Key results

• On GPT-3.5 Turbo, SCIE Zero-Shot CoT improves accuracy over Zero-Shot CoT on 9 of 10 tasks (e.g., StrategyQA 65.5 to 71.1, GSM8K 75.5 to 77.3); Date Understanding drops slightly (71.8 to 71.0).
• On GSM8K, SCIE instruction reaches 78.6% vs OPRO's 77.7% under the Q-end setting.
• OR-SCIE generalization: MultiArith with inherited causal class reaches 94.3%, above the no-SCIE baseline; aggregation on 70% of GSM8K reaches 77.9%.

Measured effects reported in the paper

Surprising findings

• Propensity score matching on the generated observational data reduced performance and was discarded.
• Removing the counterfactual instruction generation step makes SCIE accuracy fall below the unenhanced baseline.

Limitations

• Gains are small (often ~1-2 points) and no variance or significance reporting is given.
• Date Understanding degrades and the authors attribute this to bias in ATE estimation; DisambiguationQA results are inconsistent across models.
• Theoretical guarantees for the causal estimation and the instruction-optimization claim are not established; the ITE-based argument is informal.

Experimental setup

Datasets: GSM8K MultiArith StrategyQA CommonsenseQA Coin Flip Last Letter Concatenation Boolean Expressions Big Bench Hard (Causal Judgement, Date Understanding, DisambiguationQA)

Benchmarks: GSM8K MultiArith StrategyQA CommonsenseQA Coin Flip Last Letter Concatenation Boolean Expressions Causal Judgement Date Understanding DisambiguationQA fresh-gaokao-math-2023

Metrics: accuracy average treatment effect

Baselines: Zero-Shot CoT Plan-and-Solve Prompting AgentInstruct OPRO S-Learner

Builds on: APE OPRO Zero-Shot CoT Plan-and-Solve Prompting T-Learner

Closest papers by meaning

• Unveiling Causal Reasoning in Large Language Models: Reality or Mirage? (NeurIPS 2024 · similarity 0.90)
• How to Auto-optimize Prompts for Domain Tasks? Adaptive Prompting and Reasoning through Evolutionary Domain Knowledge Adaptation (NeurIPS 2025 · similarity 0.89)
• From Noise to Diversity: Random Embedding Injection in LLM Reasoning (NeurIPS 2026 · similarity 0.88)
• Causal Sufficiency and Necessity Improves Chain-of-Thought Reasoning (NeurIPS 2025 · similarity 0.88)
• Selection, Reflection and Self-Refinement: Revisit Reasoning Tasks via a Causal Lens (ICLR 2026 · similarity 0.88)
• Causal Prompting: Debiasing Large Language Model Prompting Based on Front-Door Adjustment (AAAI 2025 · similarity 0.88)