Research map / Time series / Foundation models for time series
Disentanglement Beyond Static vs. Dynamic: A Benchmark and Evaluation Framework for Multi-Factor Sequential Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the MSD benchmark for multi-factor sequential disentanglement across six video/audio/time-series datasets, with a latent exploration stage, a per-sample Koopman model (SSM-SKD), and VLM-based tagging and judging.
Problem
Sequential disentanglement has mostly been studied with two factors (static vs. dynamic); multi-factor evaluation lacks standardized datasets, protocols, and automation, and relies on fully labeled data and manual latent-to-factor matching.
Approach
A unified multi-modal benchmark with ten metrics, a post-hoc Latent Exploration Stage that automatically maps latents to factors, a Single Static Mode SKD that uses per-sample Koopman matrices and one static eigenvector, and VLMs used as zero-shot taggers and judges.
Key results
- SSM-SKD ranks first on 4 of 6 benchmarks and scores 0.95 on Sprites and 0.96 on 3D Shapes in the aggregated table.
- Applying LES to SKD raises M-Swap from 0.69 to 0.70 and M-GSample from 0.67 to 0.71.
- VLM-based evaluation rankings show almost perfect Spearman alignment with ground-truth-supervised rankings on Sprites and 3D Shapes.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SSM-SKD instead of the strongest baseline, MGP-VAE | multi-factor sequential disentanglement · Sprites | aggregated score: 0.79 ± 3e-03 -> 0.95 ± 2e-03 | Helped |
| Use SSM-SKD instead of the strongest baseline, VAE | multi-factor sequential disentanglement · 3D Shapes | aggregated score: 0.93 ± 7e-04 -> 0.96 ± 4e-04 | Helped |
| Use SSM-SKD instead of the strongest baseline, β-VAE | multi-factor sequential disentanglement · dSprites-Static | aggregated score: 0.70 ± 2e-03 -> 0.83 ± 2e-03 | Helped |
| Use SSM-SKD instead of the strongest baseline, Sparse-AE | multi-factor sequential disentanglement · dSprites-Dynamic | aggregated score: 0.68 ± 2e-03 -> 0.71 ± 1e-03 | Helped |
| Use SSM-SKD instead of the strongest baseline, β-VAE | multi-factor sequential disentanglement · dMelodies-WAV | aggregated score: 0.52 ± 2e-03 -> 0.55 ± 2e-03 | Helped |
| Apply the Latent Exploration Stage to SKD | SKD · multi-factor tasks | M-Swap: 0.69 -> 0.70 | Helped |
| Apply the Latent Exploration Stage to SKD | SKD · multi-factor tasks | M-GSample: 0.67 -> 0.71 | Helped |
| Use SSM-SKD instead of VAE | multi-factor sequential disentanglement · BMS Air Quality | aggregated score: 0.42 ± 2e-02 -> 0.42 ± 2e-03 | No clear effect |
Surprising findings
- A vanilla VAE occasionally achieves the highest score, e.g., best on 3D Shapes at 0.93.
- No method exceeds 0.85 on either dSprites variant, and existing methods fail to disentangle factors on real-world VoxCelebOne videos.
Limitations
- Existing methods, including SSM-SKD, produce blurry reconstructions and entangle background on real-world video.
- VLM-based tagging is imperfect, with lower accuracy on lighting and shirt color.
- Several methods could not be reliably reproduced from prior work.
Experimental setup
Datasets: Sprites 3D Shapes dSprites dMelodies BMS Air Quality VoxCelebOne
Benchmarks: Sprites 3D Shapes dSprites-Static dSprites-Dynamic dMelodies-WAV BMS Air Quality VoxCelebOne
Metrics: 2-Swap 2-GSample M-Swap M-GSample DCI-M DCI-C DCI-E C-Swap C-Sample GC-Sample
Baselines: Sequential VAE Sequential beta-VAE Sequential Sparse-AE MGP-VAE SKD
Builds on: SKD MGP-VAE beta-VAE disentanglement_lib DCI
Closest papers by meaning
- Sequential Disentanglement by Extracting Static Information From A Single Sequence Element (ICML 2024 · similarity 0.90)
- Graph-based Unsupervised Disentangled Representation Learning via Multimodal Large Language Models (NeurIPS 2024 · similarity 0.88)
- Factorized Diffusion Autoencoder for Unsupervised Disentangled Representation Learning (AAAI 2024 · similarity 0.88)
- Disentangled Representation Learning via Modular Compositional Bias (NeurIPS 2025 · similarity 0.88)
- Disentangled Representation Learning via Flow Matching (NeurIPS 2026 · similarity 0.87)
- RePercENT: Scaling Disentangled Representation Learning Beyond Two Modalities (NeurIPS 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.