Research map / Vision-language and multimodal / Cross-modal learning
An Information Criterion for Controlled Disentanglement of Multimodal Data
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DisentangledSSL, a two-step information-theoretic self-supervised method that learns shared and modality-specific representations, with optimality guarantees even when the Minimum Necessary Information point is unattainable.
Problem
Learning disentangled shared and modality-specific representations of multimodal data when the Minimum Necessary Information (MNI) point is unattainable, i.e. modality-specific and shared information are deeply entangled, which prior information-theoretic methods do not analyze.
Approach
A two-step procedure: first learn shared representations by maximizing I(Z;X2) minus beta times the conditional mutual information I(Z;X1|X2) (a CEB-style objective with vMF encoders and InfoNCE); then learn modality-specific representations by maximizing InfoNCE-based joint information with the other modality's shared code while penalizing overlap with the shared code via an orthogonality loss. T
Key results
- On MultiBench, DisentangledSSL achieves the best linear-probing accuracy on all five datasets, e.g. 61.60% on MUSTARD vs 58.21% for FOCAL and 77.45% on MOSEI vs 76.87% for CLIP.
- On LINCS molecule-phenotype retrieval, DisentangledSSL reaches MRR 0.163 vs 0.161 for JointOpt; on RXRX19a it reaches MRR 0.111 and top-1 accuracy 4.03% vs 3.85% for DMVAE.
- DisentangledSSL attains the highest reconstruction gain for both molecule and phenotype modalities on RXRX19a and LINCS (e.g. RG-molecule 0.153 on RXRX19a vs 0.130 for JointOpt).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DisentangledSSL instead of JointOpt | MultiBench linear probing · MIMIC | accuracy (%): 66.11 (0.64) -> 66.44 (0.31) | Helped |
| Use DisentangledSSL instead of CLIP | MultiBench linear probing · MOSEI | accuracy (%): 76.87 (0.45) -> 77.45 (0.06) | Helped |
| Use DisentangledSSL instead of JointOpt | MultiBench linear probing · MOSI | accuracy (%): 65.02 (1.96) -> 65.16 (0.81) | No clear effect |
| Use DisentangledSSL instead of JointOpt | MultiBench linear probing · UR-FUNNY | accuracy (%): 63.58 (1.45) -> 64.24 (1.54) | Helped |
| Use DisentangledSSL instead of FOCAL | MultiBench linear probing · MUSTARD | accuracy (%): 58.21 (2.21) -> 61.60 (2.61) | Helped |
| Use DisentangledSSL instead of DMVAE | molecule-phenotype retrieval · RXRX19a | Top-1 retrieval accuracy (%): 3.85(0.36) -> 4.03(0.39) | Helped |
| Use DisentangledSSL instead of JointOpt | molecule-phenotype retrieval · LINCS | Top-1 retrieval accuracy (%): 4.67(0.09) -> 4.48(0.21) | Hurt |
| Use DisentangledSSL instead of JointOpt | molecule-phenotype retrieval · LINCS | MRR: 0.161(0.001) -> 0.163(0.002) | Helped |
| Add the information bottleneck constraint (β>0) (ablation) | DisentangledSSL molecule-phenotype retrieval · RXRX19a | Top-1 retrieval accuracy (%): 3.39(0.54) -> 4.03(0.39) | Helped |
| Add the information bottleneck constraint (β>0) (ablation) | DisentangledSSL molecule-phenotype retrieval · LINCS | Top-1 retrieval accuracy (%): 4.36(0.13) -> 4.48(0.21) | Helped |
Surprising findings
- On MultiBench MOSI and MUSTARD, only shared or only modality-specific information contributes significantly to prediction, rather than their combination.
Limitations
- Modality-specific and shared representations rely on approximations (InfoNCE, vMF KL upper bound, batch-approximated orthogonal loss), so optimality is only guaranteed in theory.
- Improvements on MultiBench and retrieval are often small, with overlapping standard deviations versus baselines.
- Sufficient conditions for MNI attainability are not comprehensively characterized.
Experimental setup
Datasets: MultiBench RXRX19a LINCS synthetic data
Benchmarks: MultiBench RXRX19a LINCS synthetic simulation
Metrics: linear probing accuracy top-N retrieval accuracy mean reciprocal rank reconstruction gain R^2
Baselines: CLIP DMVAE FOCAL FactorCL SimMMDG JointOpt
Builds on: information bottleneck conditional entropy bottleneck InfoNCE CLIP FactorCL
Closest papers by meaning
- RePercENT: Scaling Disentangled Representation Learning Beyond Two Modalities (NeurIPS 2026 · similarity 0.92)
- Graph-based Unsupervised Disentangled Representation Learning via Multimodal Large Language Models (NeurIPS 2024 · similarity 0.89)
- A Closer Look at Multimodal Representation Collapse (ICML 2025 · similarity 0.88)
- Aligning Multimodal Representations through an Information Bottleneck (ICML 2025 · similarity 0.87)
- Retrieval-based Disentangled Representation Learning with Natural Language Supervision (ICLR 2024 · similarity 0.87)
- $\alpha$TC-VAE: On the relationship between Disentanglement and Diversity (ICLR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.