Research map / Probabilistic and scientific modeling / Causal discovery
Towards Understanding Extrapolation: a Causal Lens
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Formalizes extrapolation to out-of-support target samples as latent-variable identification under a minimal-change causal model, gives dense- and sparse-shift identifiability conditions, and improves MAE-TTT and TeSLA with entropy minimization and sparsity regularization.
Problem
Predicting labels for one or a few target samples lying outside the training support (extrapolation), which is ill-posed without structural assumptions linking source and target.
Approach
A latent-variable model z=[c,s] with invariant variable c carrying the label and changing variable s controlling only non-semantic attributes; extrapolation reduces to identifying c from one source distribution and one off-support target sample, with conditions on the shift being dense (Lipschitz/distance bound) or sparse (mechanistic dependence of unaffected dimensions).
Key results
- On synthetic binary classification at distance 12-30, the method reaches 0.69-0.78 accuracy versus about 0.45-0.59 for source-only and 0.36-0.54 for iMSDA.
- Adding entropy minimization to MAE-TTT raises ImageNet100-C accuracy from 59.12% to 65.09% with two steps.
- TeSLA-s with sparsity constraint lowers ImageNet-C average error from 53.1% to 50.9%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the proposed adaptation method instead of the strongest listed baseline | Synthetic classification · Dense shift, distances 12.0, 18.0, 24.0, and 30.0 | accuracy: 0.59 -> 0.78; 0.48 -> 0.69; 0.48 -> 0.72; 0.50 -> 0.72 | Helped |
| Use the proposed adaptation method instead of the strongest listed baseline | Synthetic classification · Sparse shift, distances 18.0, 24.0, 30.0, and 36.0 | accuracy: 0.54 -> 0.72; 0.54 -> 0.72; 0.56 -> 0.76; 0.54 -> 0.70 | Helped |
| Add sparsity constraints to TeSLA-s (ablation) | TeSLA-s · CIFAR10-C | error rate: 12.1 -> 11.7 | Helped |
| Add sparsity constraints to TeSLA-s (ablation) | TeSLA-s · CIFAR100-C | error rate: 37.3 -> 37.0 | Helped |
| Add sparsity constraints to TeSLA-s (ablation) | TeSLA-s · ImageNet-C | error rate: 53.1 -> 50.9 | Helped |
| Add sparsity constraints to TeSLA* (ablation) | TeSLA* · CIFAR10-C | error rate: 12.5 ± 0.04 -> 12.1 ± 0.11 | Helped |
| Add sparsity constraints to TeSLA* (ablation) | TeSLA* · CIFAR100-C | error rate: 38.2 ± 0.03 -> 38.0 ± 0.13 | Helped |
| Add sparsity constraints to TeSLA* (ablation) | TeSLA* · ImageNet-C | error rate: 55.0 ± 0.17 -> 54.5 ± 0.12 | Helped |
| Add entropy-minimization steps to MAE-TTT | MAE-TTT · ImageNet-C | accuracy (%): 45.92 -> 47.77 | Helped |
| Add 1 entropy-minimization step (ablation) | MAE-TTT · ImageNet100-C | accuracy: 59.12 ± 0.35 -> 63.99 ± 0.25 | Helped |
| Increase entropy-minimization steps from 1 to 2 (ablation) | MAE-TTT · ImageNet100-C | accuracy: 63.99 ± 0.25 -> 65.09 ± 0.23 | Helped |
| Increase entropy-minimization steps from 2 to 3 (ablation) | MAE-TTT · ImageNet100-C | accuracy: 65.09 ± 0.23 -> 65.01 ± 0.39 | No clear effect |
Surprising findings
- Domain adaptation baseline iMSDA performs near chance or worse than source-only on synthetic extrapolation, suggesting direct application can hurt.
- Under sparse shifts, classification error for TENT stays nearly constant across corruption severity, whereas large dense block shifts cause sharp collapse.
Limitations
- Global Jacobian-norm condition may be too stringent; a local smoothness condition could relax it.
- Requires an explicit learned representation space; methods without such structure benefit less.
- Multiple loss terms (reconstruction, classification, target likelihood) may need careful re-weighting.
Experimental setup
Datasets: CIFAR10-C CIFAR100-C ImageNet-C ImageNet100-C synthetic latent-variable data
Benchmarks: CIFAR10-C CIFAR100-C ImageNet-C ImageNet100-C synthetic dense/sparse shift
Metrics: classification accuracy error rate MSE
Baselines: iMSDA Only Source BN TENT SHOT TTT++ TTAC TeSLA TTT-MAE
Builds on: MAE-TTT TeSLA iMSDA SHOT
Closest papers by meaning
- Out-of-Distribution Generalization via Risk Extrapolation (REx) (ICML 2021 · similarity 0.87)
- A Theory of Independent Mechanisms for Extrapolation in Generative Models (AAAI 2021 · similarity 0.87)
- Identifying Representations for Intervention Extrapolation (ICLR 2024 · similarity 0.85)
- Concept Modulation Models: A Unified Framework for Identifiability and Extrapolation (NeurIPS 2026 · similarity 0.85)
- Identifiable Latent Polynomial Causal Models through the Lens of Change (ICLR 2024 · similarity 0.84)
- Final-Model-Only Data Attribution with a Unifying View of Gradient-Based Methods (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.