Research map / Large language models / Language, safety and interpretability
End-To-End Causal Effect Estimation from Unstructured Natural Language Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces NATURAL, LLM-based estimators that compute classical ATE estimators (IPW/OI) from conditional distributions extracted from unstructured Reddit and synthetic text reports, matching RCT ground truths within about 3 percentage points.
Problem
Causal effect estimation normally needs structured tabular observational data, which is costly to collect and structure; unstructured text (e.g., online patient forum posts) is abundant but unused for ATE estimation.
Approach
Derive ATE estimators (IPW and outcome imputation, full, hybrid, and Monte Carlo variants) that take LLM conditional distributions P(X,T,Y|R) over text reports as inputs, zero-shot, with an LLM pipeline for filtering, covariate extraction, conditional inference, and inclusion-criteria weighting.
Key results
- On real clinical settings, best NATURAL estimators fall within 3 percentage points of RCT ground-truth ATEs (e.g., N-IPW 9.06% vs 10.11% for Semaglutide vs. Tirzepatide).
- On Hillstrom, NATURAL IPW achieves RMSE 1.32 versus 2.23 for the Bag-of-Words trained baseline and 6.09 for the sentence encoder baseline (ATE 5.23% vs 6.09% ground truth).
- Ablations show inclusion-criteria filtering and pretrained LLM imputations are both crucial to performance, and performance improves with LLM scale and data quantity.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use NATURAL IPW instead of the other NATURAL estimators | NATURAL on Hillstrom and Retail Hero · Hillstrom and Retail Hero | RMSE: 1.32; 1.39 | Helped |
| Use NATURAL OI instead of NATURAL IPW | NATURAL on Semaglutide vs. Liraglutide · Semaglutide vs. Liraglutide | RMSE: 2.15 vs. 2.33 | Helped |
| Use NATURAL OI instead of NATURAL IPW | NATURAL on Erenumab vs. Topiramate · Erenumab vs. Topiramate | RMSE: 1.92 vs. 2.68 | Helped |
| Use NATURAL IPW instead of NATURAL OI | NATURAL on OnabotulinumtoxinA vs. Topiramate · OnabotulinumtoxinA vs. Topiramate | RMSE: 2.57 vs. 3.99 | Helped |
| Add inclusion-criteria filtering (ablation) | NATURAL on Semaglutide vs. Tirzepatide · Semaglutide vs. Tirzepatide | ATE prediction RMSE | Helped |
| Use pretrained-LLM imputations instead of uniform-distribution samples (ablation) | NATURAL on Semaglutide vs. Tirzepatide · Semaglutide vs. Tirzepatide | ATE prediction RMSE | Helped |
| Use larger-scale models for conditional probabilities (ablation) | NATURAL on Semaglutide vs. Tirzepatide · Semaglutide vs. Tirzepatide | ATE prediction RMSE | Helped |
| Increase the quantity of data (ablation) | NATURAL on Semaglutide vs. Tirzepatide · Semaglutide vs. Tirzepatide | ATE prediction RMSE | Helped |
| Use propensity scores estimated from LLAMA2 conditionals instead of uniform scores | NATURAL on Semaglutide vs. Tirzepatide · Semaglutide vs. Tirzepatide | Covariate balance | Helped |
| Discretize continuous covariates into roughly balanced intervals (ablation) | NATURAL estimators · Semaglutide vs. Tirzepatide | ATE prediction accuracy | Helped |
Surprising findings
- A sentence-encoder baseline collapsed to an ATE of zero on Hillstrom, having learned a constant outcome predictor.
- The method is zero-shot and needs no labeled task-specific training data, yet matches or beats trained Bag-of-Words and sentence-encoder baselines on synthetic data.
Limitations
- Estimates are only local ATEs; selection bias in who posts online limits external validity.
- Relies on LLM calibration; the pipeline cannot be formally guaranteed to satisfy the required conditional-access assumptions.
- Requires expert-specified covariates and causal assumptions; limited to binary treatments and outcomes.
Experimental setup
Datasets: Hillstrom Retail Hero Pushshift Reddit posts (migraine and diabetes subreddits) Semaglutide vs. Tirzepatide RCT Semaglutide vs. Liraglutide RCT Erenumab vs. Topiramate RCT OnabotulinumtoxinA vs. Topiramate RCT
Benchmarks: Hillstrom Retail Hero Semaglutide vs. Tirzepatide Semaglutide vs. Liraglutide Erenumab vs. Topiramate OnabotulinumtoxinA vs. Topiramate
Metrics: ATE error RMSE KL divergence
Baselines: Uncorrected Selection-biased N-IPW Bag-of-Words Sentence Encoder IPW (Structured)
Builds on: inverse propensity score weighting outcome imputation Pushshift
Closest papers by meaning
- Proximal Causal Inference With Text Data (NeurIPS 2024 · similarity 0.90)
- LLM-Driven Treatment Effect Estimation Under Inference Time Text Confounding (NeurIPS 2025 · similarity 0.88)
- Isolated Causal Effects of Natural Language (ICML 2025 · similarity 0.88)
- Data Fusion for Partial Identification of Causal Effects (NeurIPS 2025 · similarity 0.87)
- Revealing Multimodal Causality with Large Language Models (NeurIPS 2025 · similarity 0.87)
- Can Large Language Models Infer Causation from Correlation? (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.