Research map / Vision-language and multimodal / Human and embodied modeling
Revisiting Disentanglement in Downstream Tasks: A Study on Its Necessity for Abstract Visual Reasoning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
The paper empirically tests whether dimension-wise disentangled representations are necessary for abstract visual reasoning (RPMs), training 720 Stage-1 encoders (DisVAEs and BYOL) and 7200 WReN/Transformer Stage-2 reasoners, and finds informativeness correlates more with downstream accuracy than disentanglement.
Problem
Whether disentangled representations are necessary for downstream tasks, specifically abstract visual reasoning on Raven's Progressive Matrices, given prior claims of their benefits and evidence that contrastive methods achieve strong results without full disentanglement.
Approach
Uses a two-stage evaluation (frozen unsupervised encoder then reasoning model) across disentangled VAEs and general-purpose BYOL encoders, and measures rank correlation of many disentanglement and informativeness metrics with downstream accuracy; informativeness (multinomial logistic regression accuracy) is the strongest correlate, and disentanglement adds little once informativeness is matched.
Key results
- On 3DShapes, the best BYOL-WReN reached 87.1% and DisVAE-WReN 84.8%, so entangled BYOL matched or beat disentangled DisVAEs despite much lower MED scores.
- Logistic-regression informativeness had the strongest rank correlation with downstream accuracy at each training step; for BYOL on 3DShapes LR correlated at 96.6 (x100) with WReN accuracy, versus 8.1 for MIG.
- Once informativeness is matched (adjusted metrics), disentanglement metrics correlate only mildly with downstream accuracy, and they contribute marginal benefit in the first 2000 steps.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use BYOL rather than DisVAEs for the Stage-1 representation | WReN · 3DShapes | accuracy: 84.8(0.91) -> 87.1(4.68) | Helped |
| Use BYOL rather than DisVAEs for the Stage-1 representation | Transformer · 3DShapes | accuracy: 87.0(6.36) -> 88.0(2.62) | Helped |
| Use BYOL rather than DisVAEs for the Stage-1 representation | WReN · Abstract dSprites | accuracy: 68.7{(1.39)} -> 72.2{(3.11)} | Helped |
| Use BYOL rather than DisVAEs for the Stage-1 representation | Transformer · Abstract dSprites | accuracy: 66.4{(7.06)} -> 78.1{(1.75)} | Helped |
| Prefer more informative representations, measured by Logistic Regression accuracy, over more disentangled representations | Stage-1 representations for abstract visual reasoning · 3DShapes and Abstract dSprites | downstream accuracy correlation | Helped |
| Match representations for informativeness before increasing disentanglement | DisVAE-WReN pipeline · 3DShapes | downstream accuracy correlation | No clear effect |
| Use disentangled rather than general-purpose representations | Stage-1 representations for abstract visual reasoning · 3DShapes | sample efficiency | No clear effect |
Surprising findings
- General-purpose BYOL encoders with weak disentanglement (low MED) matched or outperformed disentangled DisVAEs on abstract reasoning.
- Disentanglement metrics such as MIG and SAP showed negative regression slopes against downstream accuracy for BYOL.
Limitations
- Only evaluated on abstract visual reasoning (RPM) with two synthetic datasets; generalization to other downstream tasks is untested.
- Disentanglement is restricted to the dimension-wise definition; alternative definitions were not studied.
- Limited compute restricted the scope of downstream tasks.
Experimental setup
Datasets: Abstract dSprites 3DShapes
Benchmarks: Abstract dSprites RPM (hard-mixed) 3DShapes RPM (hard-mixed)
Metrics: test accuracy Spearman rank correlation BetaVAE score FactorVAE score MIG SAP MED DCI disentanglement logistic regression accuracy linear regression NMSE
Baselines: DisVAEs (beta-VAE, AnnealedVAE, beta-TCVAE, FactorVAE, DIP-VAE) BYOL SimSiam WReN Transformer
Builds on: disentanglement_lib BYOL WReN Transformer beta-VAE
Closest papers by meaning
- Disentangling Representations through Multi-task Learning (ICLR 2025 · similarity 0.89)
- Transferring disentangled representations: bridging the gap between synthetic and real images (NeurIPS 2024 · similarity 0.88)
- Disentangled Representation Learning via Flow Matching (NeurIPS 2026 · similarity 0.87)
- Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objects (NeurIPS 2024 · similarity 0.87)
- Disentangled Representation Learning in Non-Markovian Causal Systems (NeurIPS 2024 · similarity 0.87)
- Rethinking Multi-view Representation Learning via Distilled Disentangling (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.