Research map / Probabilistic and scientific modeling / Molecules and proteins
Tripod: Three Complementary Inductive Biases for Disentangled Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Combines finite scalar quantization, kernel-based latent multiinformation, and a normalized Hessian penalty into one autoencoder (Tripod) for unsupervised disentangled representation learning, reaching state-of-the-art InfoMEC/DCI on four synthetic/real image benchmarks.
Problem
Unsupervised disentangled representation learning is underspecified, and naively combining three existing inductive biases (latent quantization, latent multiinformation regularization, and data-generating mixed-derivative regularization) in an autoencoder performs poorly due to optimization difficul
Approach
Make each of three complementary inductive biases work in an autoencoder with lightest-touch adaptations: finite scalar quantization (fixed codebook) instead of VQ, kernel density estimation-based multiinformation for deterministic encoders with std-scaled bandwidths, and a normalized Hessian penalty invariant to latent and activation rescaling.
Key results
- Tripod achieves aggregate InfoMEC of (0.78, 0.59, 0.90) and DCI of (0.64, 0.57, 0.93) across four benchmarks, state-of-the-art versus beta-TCVAE and QLAE.
- Naive Tripod is virtually indistinguishable from QLAE in aggregate, showing the proposed adaptations are necessary.
- Removing the NHP leg causes a significant drop in modularity and compactness; FSQ with finer quantization (12^2 values) incurs the largest penalty in ablations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Tripod instead of β-TCVAE | Disentangled representation learning · Shapes3D, MPI3D, Falcor3D, and Isaac3D | modularity | Helped |
| Use Tripod instead of QLAE | Disentangled representation learning · Shapes3D, MPI3D, Falcor3D, and Isaac3D | modularity | Helped |
| Use Tripod instead of naive Tripod | Disentangled representation learning · Shapes3D, MPI3D, Falcor3D, and Isaac3D | disentanglement metrics | Helped |
| Add normalized Hessian penalty (NHP) (ablation) | Tripod · Shapes3D, MPI3D, Falcor3D, and Isaac3D | modularity | Helped |
| Add kernel-based latent multiinformation (KLM) (ablation) | Tripod · Shapes3D, MPI3D, Falcor3D, and Isaac3D | modularity | Helped |
| Use 12 quantized values per dimension instead of 12^2 (ablation) | Tripod · Shapes3D, MPI3D, Falcor3D, and Isaac3D | disentanglement | Helped |
| Add NHP to QLAE + KLM | Training iteration · Profiling study | runtime: about 2.5 | Hurt |
Surprising findings
- Naive combination of the three existing inductive biases fails to yield significant benefits over QLAE alone.
- Finer quantization (12^2 values) was the most damaging ablation, more than removing either KLM or NHP.
Limitations
- Evaluation limited to image datasets with ground-truth sources, which are synthetic or robotic, not real-world data.
- Sensitive to the degree of quantization, which must be tuned using labels.
- NHP training iteration runtime is about 2.5x higher due to extra decoder forward passes.
Experimental setup
Datasets: Shapes3D MPI3D Falcor3D Isaac3D
Benchmarks: Shapes3D MPI3D Falcor3D Isaac3D
Metrics: InfoM InfoC InfoE D C I
Baselines: beta-TCVAE QLAE naive Tripod
Builds on: QLAE beta-TCVAE Hessian penalty finite scalar quantization Silverman's rule of thumb
Closest papers by meaning
- Disentanglement via Latent Quantization (NeurIPS 2023 · similarity 0.92)
- Orthogonality-Enforced Latent Space in Autoencoders: An Approach to Learning Disentangled Representations (ICML 2023 · similarity 0.91)
- Disentangled Representation Learning via Modular Compositional Bias (NeurIPS 2025 · similarity 0.89)
- Diffusion Model with Cross Attention as an Inductive Bias for Disentanglement (NeurIPS 2024 · similarity 0.88)
- Disentangled Representation Learning via Flow Matching (NeurIPS 2026 · similarity 0.88)
- Factorized Diffusion Autoencoder for Unsupervised Disentangled Representation Learning (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.