Research map / Probabilistic and scientific modeling / Causal discovery
Sanity Checking Causal Representation Learning on a Simple Real-World System
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Builds a real optical "light tunnel" testbed with known causal inputs and evaluates three CRL method families (contrastive CRL, multiview CRL, CITRIS) on real and synthetic-ablation data, finding they largely fail.
Problem
Causal representation learning methods are mostly validated on synthetic data built under their own assumptions; it is unclear whether they work on real physical systems where the causal factors are known.
Approach
A controlled optical experiment (light tunnel with RGB LEDs and two polarizers) whose inputs serve as ground-truth causal factors, used as a real-world sanity check, plus a deterministic synthetic simulator ablation to separate mixing-function misspecification from optimization failure.
Key results
- Contrastive CRL performs well on the synthetic ablation but fails on real data, with unstable latent graphs across five random initializations.
- Multiview CRL and CITRIS fail on both real and synthetic-ablation data, with low diagonal R^2 scores and relatively high off-diagonal values.
- A supervised MLP recovers the causal factors from real images with R^2 of 0.976, indicating the information is present in the data.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace real mixing data with deterministic simulated data (ablation) | Contrastive CRL · Light tunnel benchmark | MCC | Helped |
| Replace deterministic simulated data with real light tunnel data (ablation) | Contrastive CRL · Light tunnel benchmark | MCC | Hurt |
| Replace deterministic simulated data with real light tunnel data (ablation) | Contrastive CRL · Light tunnel benchmark | SHD | Hurt |
| Use the view-1/view-2 intersection, whose sensor view violates injectivity | Multiview CRL · Light tunnel real data, views 1 and 2 | R² | Helped |
| Use view-1/view-3 or view-1/view-4 intersections | Multiview CRL · Light tunnel real data, views 1 and 3; views 1 and 4 | R² | Hurt |
| Replace real mixing data with deterministic simulated data (ablation) | Multiview CRL · Light tunnel benchmark | R² | No clear effect |
| Replace real mixing data with deterministic simulated data (ablation) | CITRIS-VAE · Light tunnel benchmark | R² | No clear effect |
| Replace real mixing data with deterministic simulated data (ablation) | CITRIS-VAE · Light tunnel benchmark | Spearman correlation | No clear effect |
| Use an autoregressive prior without normalizing flows (ablation) | CITRIS-VAE · Light tunnel benchmark | final score | No clear effect |
| Set the model latent dimension to match the ground truth (ablation) | CITRIS-VAE · Light tunnel benchmark | final score | No clear effect |
| Fix latent-to-factor assignments instead of learning them (ablation) | CITRIS-VAE · Light tunnel benchmark | final score | No clear effect |
| Train for 500 instead of 250 epochs (ablation) | CITRIS-VAE · Light tunnel benchmark | final score | No clear effect |
Surprising findings
- Most CRL methods fail even on the simple deterministic synthetic ablation, indicating a reproducibility problem.
- Multiview CRL performs better on a misspecified (non-injective) view intersection than on injective views with identity mixing.
Limitations
- Only tests mixing-function assumptions; latent causal model assumptions cannot be evaluated with this setup.
- Only three representative methods evaluated, chosen partly by public code availability.
- Failures may stem from untested architectures and hyperparameters, so optimization failures cannot be excluded.
Experimental setup
Datasets: lt_crl_benchmark_v1 (light tunnel data) synthetic ablation via deterministic simulator
Benchmarks: lt_crl_benchmark_v1 (real light tunnel data) deterministic synthetic ablation
Metrics: mean correlation coefficient (MCC) structural Hamming distance (SHD) R^2 score Spearman rank correlation
Baselines: Contrastive CRL Multiview CRL CITRIS
Builds on: Contrastive CRL Multiview CRL CITRIS
Closest papers by meaning
- CausalVerse: Benchmarking Causal Representation Learning with Configurable High-Fidelity Simulations (NeurIPS 2025 · similarity 0.92)
- Causal Representation Learning Made Identifiable by Grouping of Observational Variables (ICML 2024 · similarity 0.90)
- Learning Linear Causal Representations from General Environments: Identifiability and Intrinsic Ambiguity (NeurIPS 2024 · similarity 0.88)
- A Sparsity Principle for Partially Observable Causal Representation Learning (ICML 2024 · similarity 0.87)
- Causal Representation Learning via Counterfactual Intervention (AAAI 2024 · similarity 0.87)
- Unifying Causal Representation Learning with the Invariance Principle (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.