Research map / Label-efficient and robust learning / Recognition and long-tail
Transferring disentangled representations: bridging the gap between synthetic and real images
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Studies transferring disentangled representations learned with weak supervision on synthetic datasets to real image datasets via unsupervised VAE fine-tuning, and proposes OMES, a classifier-free intervention-based disentanglement metric.
Problem
Disentangled representation learning is mostly validated on synthetic data and struggles on real images due to correlated factors, resolution, and lack of factor labels; it is unclear whether disentanglement transfers from synthetic sources to real targets.
Approach
Learn beta-VAE representations with weak supervision (Ada-GVAE) on a labelled synthetic source, then transfer and unsupervisedly fine-tune on an unlabelled target (synthetic or real); additionally propose OMES, a classifier-free intervention-based metric using correlation between dimensions and factors across paired images that differ in one factor.
Key results
- Fine-tuning on the target gives gains in explicitness; e.g. Color-dSprites to Shapes3D increases GBT accuracy on all factors from 53.0% to 40.0%-equivalent pruned comparisons, with full-representation accuracy improving by 13.6 points (dSprites to Shapes3D).
- OMES correlates highly with MIG and DCI and mildly with BetaVAE and FactorVAE scores on the 5400 pre-trained models from prior work.
- Explicitness is usually preserved when transferring from synthetic to real data, while modularity and compactness degrade, with fine-tuning giving limited benefit to these.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Fine-tune the transferred model on the target dataset | Ada-GVAE transferred to synthetic targets · Shapes3D | All FoVs accuracy (%): +13.6 | Helped |
| Fine-tune the transferred model on the target dataset | Ada-GVAE transferred to a real target · Coil | All FoVs accuracy (%): +10.4 | Helped |
| Fine-tune the transferred model on the target dataset | Ada-GVAE transferred to a real target · RGBD | All FoVs accuracy (%): +1.3 | Helped |
| Fine-tune the transferred model on the target dataset | Ada-GVAE transferred from Shapes3D · Isaac3D | All FoVs accuracy (%): +16.7 | Helped |
| Use only the OMES-selected single dimension instead of the full representation (ablation) | Ada-GVAE transfer from dSprites · Shapes3D | All FoVs accuracy (%): 53.0 -> 40.0 | Hurt |
| Use only the OMES-selected single dimension instead of the full representation (ablation) | Ada-GVAE transfer from C-dSprites · Coil | All FoVs accuracy (%): 24.5 -> 14.9 | Hurt |
| Use only the OMES-selected single dimension instead of the full representation (ablation) | Ada-GVAE transfer from C-dSprites · RGBD | All FoVs accuracy (%): 50.1 -> 28.5 | Hurt |
| Use only the OMES-selected single dimension instead of the full representation (ablation) | Ada-GVAE transfer from Shapes3D · Isaac3D | All FoVs accuracy (%): 51.3 -> 42.1 | Hurt |
Surprising findings
- Using a simplified binary Coil as source causes only a limited reduction in explicitness despite a large complexity gap with the target.
- Using a real DRL-compliant dataset as source gave no benefit, and fine-tuning was not particularly effective.
Limitations
- Study restricted to VAE-based disentanglement approaches; generalization to vector-based methods not verified.
- OMES requires partially known factors and paired samples differing in one factor.
- Transfer effectiveness depends on unquantified distance between source and target datasets.
Experimental setup
Datasets: dSprites Noisy-dSprites Color-dSprites Noisy-Color-dSprites Shapes3D Isaac3D Coil100 RGBD-Objects SmallNORB Cars3D
Benchmarks: Color-dSprites Noisy-dSprites Shapes3D Isaac3D Coil100 RGBD-Objects
Metrics: GBT classification accuracy (full and pruned representation) OMES (OS, MES) DCI MIG rank correlation with ELBO/reconstruction loss
Baselines: DCI MIG BetaVAE score FactorVAE score Modularity
Builds on: beta-VAE Ada-GVAE DCI MIG
Closest papers by meaning
- On the Transfer of Disentangled Representations in Realistic Settings (ICLR 2021 · similarity 0.93)
- Learning Disentangled Representation by Exploiting Pretrained Generative Models: A Contrastive Learning View (ICLR 2022 · similarity 0.91)
- Disentangled Representation Learning via Flow Matching (NeurIPS 2026 · similarity 0.89)
- Disentangled Representation Learning via Modular Compositional Bias (NeurIPS 2025 · similarity 0.88)
- Revisiting Disentanglement in Downstream Tasks: A Study on Its Necessity for Abstract Visual Reasoning (AAAI 2024 · similarity 0.88)
- Synergy Between Sufficient Changes and Sparse Mixing Procedure for Disentangled Representation Learning (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.