Research map / Time series / Causal and generative time series
Learning diverse causally emergent representations from time series data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an end-to-end differentiable method that learns macroscopic features of multivariate time series maximising a causal-emergence criterion (Ψ) estimated with SMILE, extended with a diversity penalty to learn multiple independent emergent features, and applies it to synthetic bit-strings, Game of Life and brain recordings (ECoG, MEG, fMRI).
Problem
Identifying macroscopic variables with emergent properties in time series data is hard because relevant variables are rarely known a priori; existing emergence metrics are not directly usable for representation learning.
Approach
Use a differentiable sufficient criterion for causal emergence (Ψ, with a redundancy-adjusted variant ΨA) as the training objective for a representation network, estimating all mutual information terms with the SMILE lower bound, plus an adversarial-style critic that penalises mutual information with a previously learned emergent feature to obtain diverse features.
Key results
- On the bit-string dataset the full emergence objective recovers the ground-truth Ψ and learns the parity bit, whereas prediction-only MI fails to learn any emergent feature.
- Adding a diversity penalty yields a second independent emergent feature (XOR 'bonus bit') that the single-objective model learned in only 9% of runs, while the diversity term reliably recovers both.
- Learned emergent features on ECoG, MEG and fMRI have positive post-hoc ΨA (1.2 ± 0.19, 2.6 ± 0.52, 3.3 ± 0.44), and all experiments run on a single A10G GPU in under two hours.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the full emergence objective instead of predictive mutual information alone (ablation) | Representation learner on the bit-string dataset · Bit-string dataset | emergent feature recovery | Helped |
| Remove the marginal mutual information terms (ablation) | Emergence objective on the bit-string dataset · Bit-string dataset | emergent feature recovery | Hurt |
| Add the diversity penalty between learned features (ablation) | Representation learner trained for multiple emergent features · Bit-string dataset | feature diversity: 91%; 9% | Helped |
| Remove the predictive MI term from the Ψ objective (ablation) | Representation learner with skip connections · Bit-string dataset | emergence | Hurt |
| Add skip connections to the representation learner (ablation) | Causally emergent representation learning on ECoG · ECoG | emergent feature recovery | Helped |
| Remove the macroscopic MI term while retaining skip connections (ablation) | Causally emergent representation learning on ECoG · ECoG | emergent feature recovery | No clear effect |
| Use an RNN trained with MSE instead of the emergence learner | Representation learning on the bit-string dataset · Bit-string dataset | post-hoc ΨA: 0 | Hurt |
| Combine the RNN hidden state with the learned emergent feature | RNN and emergence learner · Bit-string dataset | mutual information with future system state | Helped |
| Use the emergence objective rather than prediction-only training | Representation learner on the Game of Life dataset · Conway's Game of Life | glider state accuracy: 35% versus 53% | Helped |
| Use a standard MLP encoder instead of the emergence learner | Representation learning on fMRI · fMRI | post-hoc ΨA: −28 ± 5 | Hurt |
Surprising findings
- A standard RNN trained with MSE and an MLP autoencoder learned no emergent features (ΨA ≈ 0 and -28 ± 5 respectively), while concatenating the emergent feature with the RNN hidden state improved prediction.
- Using skip connections, emergent features on ECoG were obtained even when the macroscopic MI term was removed from the objective.
Limitations
- Training is unstable due to the adversarial interplay between macroscopic and microscopic critics; success requires a post-hoc ΨA check and architectural tricks (skip connections, critic pretraining).
- Ψ is only a sufficient, not necessary, condition for emergence and can be negatively biased by redundancy; the adjusted ΨA is only heuristic.
- Results are shown on simple synthetic data and a few brain datasets without statistical significance testing or comparison to other emergence metrics.
Experimental setup
Datasets: synthetic bit-string dataset Conway's Game of Life simulations ECoG macaque dataset (Chao et al.) MEG pharmaco-MEG placebo dataset HCP fMRI (100 participants, Schaefer atlas)
Benchmarks: synthetic bit-string dataset (ground-truth Ψ) Conway's Game of Life glider state/position decoding ECoG MEG fMRI
Metrics: Ψ (emergence objective) post-hoc ΨA decoding accuracy predictive mutual information mutual information between features
Baselines: prediction-only MI objective (Deep InfoMax style) RNN hidden state MLP encoder-decoder
Builds on: causal emergence (Rosas et al. 2020 / Ψ criterion) SMILE mutual information estimator Deep InfoMax PhiID
Closest papers by meaning
- Learning Linear Causal Representations from General Environments: Identifiability and Intrinsic Ambiguity (NeurIPS 2024 · similarity 0.86)
- Towards Identifiability of Hierarchical Temporal Causal Representation Learning (NeurIPS 2025 · similarity 0.86)
- Disentangling Representations through Multi-task Learning (ICLR 2025 · similarity 0.86)
- CaRiNG: Learning Temporal Causal Representation under Non-Invertible Generation Process (ICML 2024 · similarity 0.85)
- On the Identification of Temporal Causal Representation with Instantaneous Dependence (ICLR 2025 · similarity 0.85)
- A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Language (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.