Research map / Vision-language and multimodal / Multimodal LLMs
Graph-based Unsupervised Disentangled Representation Learning via Multimodal Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GEM, which combines a β-VAE disentanglement branch with GPT-4o-scored attribute interrelations in a bidirectional weighted graph refined by a GNN, for unsupervised relation-aware disentanglement on CelebA and LSUN.
Problem
Standard VAE-based disentangled representation learning assumes factors are independent and fully disentangleable without supervision, which fails on complex real-world data where attributes are correlated; existing structured methods need supervision and model only binary, unidirectional relations.
Approach
Uses a β-VAE to extract factors as graph nodes, prompts an MLLM (GPT-4o) to score attributes and rank bidirectional pairwise impact scores via Somers' D, and refines the weighted adjacency matrix with a GNN-based graph learner, feeding relation-aware latents to the decoder.
Key results
- GEM achieves FID 46.0 on CelebA vs 70.7 for DEAR and 53.3 for VAE, and lower FID on LSUN-horse (101.0) and LSUN-bedroom (125.4).
- GPT-4o achieves 88.4% average attribute-scoring accuracy on CelebA with a 0.25% zero-scoring rate, outperforming GPT-4V and GLM-4.
- GEM (Single) uses 44.7M params and 51.5 s training time, comparable to FactorVAE (55.9M, 63.9 s) and more efficient than DEAR (53.4M, 91.8 s).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GEM instead of the strongest listed baseline | CelebA reconstruction · CelebA | FID: 53.3 ± 0.6 -> 46.0 ± 0.1 | Helped |
| use GEM instead of the strongest listed baseline | CelebA reconstruction · CelebA | KID × 10^3: 51.4 ± 0.4 -> 48.3 ± 0.2 | Helped |
| use GEM instead of the strongest listed baseline | LSUN-horse reconstruction · LSUN-horse | FID: 136.4 ± 1.6 -> 101.0 ± 1.1 | Helped |
| use GEM instead of the strongest listed baseline | LSUN-horse reconstruction · LSUN-horse | KID × 10^3: 113.7 ± 0.9 -> 65.5 ± 1.7 | Helped |
| use GEM instead of the strongest listed baseline | LSUN-bedroom reconstruction · LSUN-bedroom | FID: 177.6 ± 3.5 -> 125.4 ± 1.2 | Helped |
| use GEM instead of the strongest listed baseline | LSUN-bedroom reconstruction · LSUN-bedroom | KID × 10^3: 157.8 ± 2.3 -> 76.1 ± 1.1 | Helped |
| replace β-VAE with vanilla VAE (ablation) | GEM · CelebA | disentanglement capability | Hurt |
| replace β-VAE with vanilla VAE (ablation) | GEM · CelebA | reconstruction quality | Helped |
| remove the GNN-based graph learner (ablation) | GEM · CelebA | relation determination | Hurt |
| remove the graph learner and relation initialization (ablation) | GEM · CelebA | interrelation learning | Hurt |
| remove adversarial training and use only the standard VAE loss (ablation) | GEM · CelebA | reconstruction quality | Hurt |
| add relation discovery and refinement modules (ablation) | GEM (Single) · Relation-aware model efficiency | training time: 51.5 -> 78.9 | Hurt |
Surprising findings
- The paper reports that the VAE baseline obtains a lower KID than GEM on CelebA (51.4 vs 48.3 is not lower; GEM's KID is best there), but the VAE's FID on LSUN-horse (172.8) is better than β-TCVAE's; the rest of the table is consistent with GEM being best on most metrics.
Limitations
- Reconstruction quality trades off against disentanglement via the β coefficient and falls short of GAN/diffusion generative models.
- Artifacts and degraded results on cluttered LSUN scenes at 64x64 resolution.
- Relies on a closed-source MLLM (GPT-4o) for relation scoring, with reliability validated only against CelebA labels.
Experimental setup
Datasets: CelebA LSUN
Benchmarks: CelebA LSUN-horse LSUN-bedroom
Metrics: FID KID Params (M) GFLOPs Memory (M) Training time (s)
Baselines: VAE beta-VAE beta-TCVAE FactorVAE DEAR
Builds on: beta-VAE FactorVAE DEAR Somers' D graph convolutional network
Closest papers by meaning
- Disentangled Representation Learning via Flow Matching (NeurIPS 2026 · similarity 0.90)
- Retrieval-based Disentangled Representation Learning with Natural Language Supervision (ICLR 2024 · similarity 0.90)
- An Information Criterion for Controlled Disentanglement of Multimodal Data (ICLR 2025 · similarity 0.89)
- Disentangled Representation Learning via Modular Compositional Bias (NeurIPS 2025 · similarity 0.89)
- Enriching Disentanglement: From Logical Definitions to Quantitative Metrics (NeurIPS 2024 · similarity 0.88)
- Factorized Diffusion Autoencoder for Unsupervised Disentangled Representation Learning (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.